Index Says Success, Search Says Nothing
A diagnostic guide for PeopleSoft Global Search on OpenSearch
“Why did the search definition stop speaking to Integration Broker? It developed trust issues.”
Every PeopleSoft administrator meets this ticket eventually. The index build finished. Crawl Status says Success. The run control is clean, the cluster is green, and the Global Search bar returns absolutely nothing. According to every component that reports on itself, nothing is wrong.
This one surfaces in the HEUG Technical community every few weeks, and almost always after one of three events: a PeopleTools 8.6x upgrade, an OpenSearch cutover, or a database refresh or clone. The cause is rarely exotic. In practice it is one of six things, and five of them live on the query side of the pipeline rather than the indexing side.
A quick reality check: a successful index build does not prove that search works. It proves that documents were written. Whether anyone is allowed to read them back is an entirely separate conversation between OpenSearch, Integration Broker and your security configuration. Call it Schrodinger’s index - the process succeeded and the results do not exist.
This guide covers the mental model, a five-minute triage sequence, the six usual causes with the exact pages to check, the errors you are likely to meet, and where the logs live. Navigation paths follow PeopleTools 8.62.
Start with the mental model, not the logs
Search Framework is a relay race, not a single process. The index build exercises the first few runners. A user’s search query is run entirely by the last few. That distinction is the whole article.
|
BUILD TIME - what a green index actually proves
|
QUERY TIME - where empty results are created
|
|
Stage 1. PS Query or Connected Query returns rows
Stage 2. Search definition, including its Security tab
Stage 3. Definition deployed to OpenSearch
Stage 4. Schedule Search Index writes the documents, through Process Scheduler and Integration Broker, or through Direct Transfer
|
Stage 5. User types into the Global Search bar
Stage 6. orcl-security-plugin authenticates that user
Stage 7. Security callback to PeopleSoft through the REST ListeningConnector
Stage 8. Results filtered by the returned entitlements, then displayed
|
|
Failure here is loud: a failed crawl, a red process instance, an error in the log.
|
Failure here is silent: every component reports success and the user still sees nothing.
|
The practical consequence is simple. If the index build is green and search is empty, spend your first ten minutes on the right-hand column. Re-running the crawl is the most popular response to this problem and the least likely to fix it.
The five-minute triage
Run these six checks, in this order, before you open a support case. Between them they resolve or correctly classify the large majority of indexed-but-empty reports.
|
Check
|
Where
|
What it proves
|
|
Ping
|
Search Instance Properties
|
Host, port, and that OpenSearch is alive. A healthy result reports the engine version back to you.
|
|
Test Login
|
Search Instance Properties
|
The search engine account credentials.
|
|
Proxy Login
|
Search Instance Properties
|
The trusted entity used on every search request. It is a different credential from the one above, and it fails separately.
|
|
Validate
|
Call Back Properties section of Search Instance Properties
|
The callback URL, plus whether the PTSF service operations, handlers and routers are active, whether the alias names match, and whether service operation security is set. The most skipped check, and the most frequent answer.
|
|
Count
|
OpenSearch Interact, Service Type: Count
|
Whether documents actually exist in the index. Crawl status does not tell you this.
|
|
No Security bisect
|
Security tab of one search definition
|
Which half of the problem you have. If results appear with security off, the fault is in the callback or in the security attributes, not in the index.
|
Check six is a diagnostic, not a fix. Leaving a definition on No Security in production is not a solution, it is a different ticket. And if the count in check five comes back as zero, skip straight to Cause 6 below: your documents never arrived, and no amount of security tuning will conjure them.
Match the symptom to the cause
Treat this as a routing table rather than a diagnosis. It tells you which section to read first.
|
What you observe
|
Most likely cause
|
Start here
|
|
Search worked before an upgrade, refresh or clone and stopped afterwards
|
The callback URL, user or password moved, or deployed definitions still carry the old values
|
Cause 1
|
|
Administrators get results, ordinary users get nothing
|
Security callback or the security attributes behind document-level security
|
Cause 2, then Cause 3
|
|
Nobody gets results, but Count shows documents in the index
|
Callback configuration, or certificate trust between the engine and PeopleSoft
|
Cause 1, then Cause 4
|
|
The OpenSearch log shows an SSL handshake failure raised by the Oracle auth plugin
|
OpenSearch does not trust the PeopleSoft certificate, so the callback never completes
|
Cause 4
|
|
A definition will not let you edit its attributes, or will not undeploy
|
It is still deployed on another search instance, or PeopleSoft and the server disagree about its state
|
Cause 5
|
|
Count returns zero documents
|
Indexing never actually delivered anything, whatever the crawl status says
|
Cause 6
|
Cause 1. Callback URL drift after an upgrade, refresh or clone
This is the most common cause by a wide margin, and the most satisfying to fix.
The callback is how OpenSearch asks PeopleSoft a question at query time. It is configured in the Call Back Properties section of Search Instance Properties, and the URL follows this shape: http(s)://server:port/PSIGW/RESTListeningConnector/node
That URL has to agree with two other places: the target location defined under Configuration, Setup Target Location, and the Content URI and Portal URI values on each local node under Configuration, Setup Local Nodes. A refresh is very good at bringing production data into test. It is equally good at bringing production URLs into test, which is how a test environment ends up politely asking production for permission.
Now the part that catches people. Changing a callback property does not update the definitions already deployed to the search engine. They keep the old URL, the old user and the old password. After any change to the callback URL, user name or password, click Update deployed definitions on Search Instance Properties. Skip that step and your index stays perfectly healthy while every query dies on arrival.
Do not take it on trust either. On Deploy/Delete Object, select your deployed definitions, click Report Sync Issues, then open View Report. The compare report lists every property where PeopleSoft and the search server disagree, and the callback URL is one of them. If the report says the definition is unavailable on the server, use Reset Definition for Deploy and deploy it again.
If you cloned the environment, or the application and web tier changed while the database stayed put, namespace aliasing is the supported way to keep using the indexes you have already built. Map the database name, node name and node URI of the new environment back to the original.
Cause 2. The callback user cannot answer
The callback user is not just a login. It needs the PTPT3300 permission list and both the Search Server and Search Query Administrator roles, and it has to exist as an active user profile on the system named in the callback URL. It should be a system account rather than an interactive one.
After a refresh, check the boring things first: a password reset or expiry, a locked account, roles stripped by a security cleanup script. If the search engine also reads attachments from an FTP location, this account needs access to that as well.
Search does not fail loudly when the callback user is locked out. It simply becomes very discreet.
Cause 3. Document-level security returns an empty entitlement set
Search Framework offers two levels of authorisation. Source-level security decides whether a user may see results from a search definition at all. Document-level security decides which individual results they see, using security attributes captured during the crawl plus an application class that returns that user’s runtime values at query time.
If the application class returns an empty list, OpenSearch filters honestly and correctly down to nothing. The engine is not malfunctioning. It is following instructions, and the instructions say show this person nothing.
So confirm three things. The application class is referenced on the Security tab of the definition. It actually returns values for your test user. And the definition passes the Audit on Deploy/Delete Object, which checks that the underlying query exists and is permissioned, that the indexed fields exist, that the security application class exists, and that the definition belongs to a search category.
Then raise the Integration Broker message log level to 5, run the search again, and confirm whether a security callback happened at all and what it returned. Empty or wrong values point back at the application class or at the user’s data permissions, not at search.
Cause 4. Certificate trust, in the direction everyone forgets
Everyone remembers to import the OpenSearch CA certificate into the PeopleSoft truststore with pskeymanager so that PeopleSoft trusts the engine. The security callback runs in the opposite direction, and that is the half that gets missed.
If the callback URL is https and SSL is not configured on OpenSearch, the PeopleSoft root CA has to be present in the OpenSearch truststore, with the truststore path and password set in opensearch.yml. The tell-tale sign is an SSL handshake exception with a PKIX path building failure, logged by the Oracle authentication plugin filter on the engine side. Indexing is completely unaffected, which is precisely why the index looks fine.
If OpenSearch sits behind a load balancer that terminates SSL, set the orclssl.callback property to false in opensearch.yml.
One more rule that catches projects mid-flight: the configuration has to be all SSL or all non-SSL. Mixing a secure component with an insecure one is not supported, however reasonable it looks on a diagram.
Cause 5. You are querying a different instance than the one you indexed
Multiple search instances are supported, and they are prioritised. The instance with the lowest priority number wins and becomes the default, which means a leftover Elasticsearch instance from your migration is still a participant in the conversation.
Two symptoms give this away. A search definition that refuses to let you edit its attributes, complaining that it is already deployed, is usually deployed on a different instance. An undeploy that fails with a not found error means PeopleSoft believes the definition is deployed while the server has no idea what you are talking about; Report Sync Issues followed by Reset Definition for Deploy clears it.
If you want a named person to test a new instance before cutover, Search Instance by User overrides the priority for that one user, which is far safer than changing the priority for everybody.
And when the old engine is genuinely finished with, ping it, prove it is dead, and take it out of the priority list. A decommissioned search engine that still answers the phone is not decommissioned.
Cause 6. The documents never actually arrived
If Count came back at zero, the documents never arrived and the crawl status was being optimistic.
Start by working out which transport you are using. If Direct Transfer is enabled, Integration Broker is bypassed entirely and the Indexing Metrics dashboard in OpenSearch Dashboards is where errors surface. If it is not enabled, Integration Broker carries documents without attachments, so the queues matter.
On the Asynchronous Services page, set Queue Level to Sub Contract and look at PTSF_ES_SEND_Q. A status of New usually means the Pub Sub domain is not active, which you confirm on the Domain Status page. A status of Error means read the message: if the engine was simply unavailable at the time, resubmit the transaction.
The Schedule Search Index page is more helpful than most people realise. It offers View Error, a link to the message log, and a link into asynchronous service monitoring. Use those before guessing.
At the cluster level, two requests answer most questions. The first is the cat indices request, which shows which indexes exist and how many documents each one holds. The second is the cat plugins request, which confirms that the Oracle security plugin is loaded on every node. That second one matters more than it sounds, because a node that joined the cluster without the plugin is a genuinely confusing failure mode.
Finally, check your replica count. Replicas must be fewer than the number of nodes. Set replicas equal to or above the node count and indexing fails with an unavailable shards exception about write consistency, which reads like a disaster and is really just arithmetic.
Errors you will actually see
Paraphrased rather than quoted verbatim, because the exact wording varies by release and patch level.
|
Message or symptom
|
What it means
|
What to do
|
|
Ping fails, or Integration Gateway reports that it could not contact the external system
|
No connectivity from PeopleSoft to the engine
|
Check host and port, confirm the engine is running, then check the Integration Broker gateway and node configuration
|
|
An authentication plugin exception reporting an invalid user or password, with HTTP status 401
|
The proxy credentials are wrong
|
Re-enter the proxy user and password, then confirm with the Proxy Login button
|
|
An SSL handshake exception with a PKIX path building failure, raised by the Oracle auth plugin filter in the OpenSearch log
|
OpenSearch does not trust the PeopleSoft certificate, so the security callback fails while indexing carries on happily
|
Import the PeopleSoft root CA into the OpenSearch truststore and set the truststore path and password in opensearch.yml
|
|
An unavailable shards exception mentioning write consistency and quorum
|
Replicas are configured at or above the number of nodes
|
Reduce the replica count on Search Options to fewer than the number of nodes
|
|
A message that the search definition is already deployed, so some fields are read only
|
It is still deployed on another search instance
|
Check every search instance on Deploy/Delete Object and undeploy where appropriate
|
|
Security index delete failed, or undeployment failed, with a not found error
|
PeopleSoft thinks the definition is deployed and the server does not have it
|
Run Report Sync Issues, then use Reset Definition for Deploy in the report column
|
|
Index creation failed with a validation exception during deploy
|
The cluster has reached its soft limit on shards per node
|
Raise the maximum shards per node setting on the cluster, then deploy again
|
|
Index schedule status shows success but no search results appear
|
The security callback is not responding with usable values, or the user has no applicable entitlements
|
Prove the document count first, then work through Causes 1 to 4 in order
|
Where to look when the pages stop helping
All PeopleSoft paths below start from PeopleTools, then Search Framework.
|
What you need
|
Where to find it
|
|
Search instance settings, Ping, Test Login, Proxy Login, Validate, Update deployed definitions and OpenSearch Interact
|
Search Admin Activity Guide, Configuration, Search Instance
|
|
Target location for services
|
Search Admin Activity Guide, Configuration, Setup Target Location
|
|
Local node Content URI and Portal URI values
|
Search Admin Activity Guide, Configuration, Setup Local Nodes
|
|
Deploy, undeploy, Report Sync Issues, Audit and Index Settings
|
Search Admin Activity Guide, Administration, Deploy/Delete Object
|
|
Index runs, View Error, message log and asynchronous monitoring
|
Search Admin Activity Guide, Administration, Schedule Search Index
|
|
Queue state for PTSF_ES_SEND_Q
|
Search Admin Activity Guide, Administration, Asynchronous Services, with Queue Level set to Sub Contract
|
|
Pub Sub domain state
|
Search Admin Activity Guide, Administration, Domain Status
|
|
Instance priority, and per-user instance overrides
|
Administration, Prioritize Search Instances, and Administration, Search Instance by User
|
|
Namespace aliases after a clone or a middle-tier change
|
Administration, Custom Namespace Settings
|
|
Engine-side errors, including callback failures
|
The OpenSearch log directory, or the path configured in opensearch.yml, plus the Integration Gateway message log in the PSIGW web application folder on the web server
|
One caution on log levels. Integration Broker message log level 5 is invaluable while you are hunting a callback, and expensive if you forget it. Turn it back down afterwards. Level 5 is a diagnostic setting, not a lifestyle.
A practical checklist
|
When
|
What to check
|
|
After every refresh, clone or upgrade
|
Reconfirm the target location. Reconfirm the local node Content URI and Portal URI. Reconfirm the callback URL, user and password, then click Validate. Click Update deployed definitions. Run Report Sync Issues across all deployed definitions and resolve or reset anything that is out of step.
|
|
Before you tell users search is ready
|
Run a Count check on two or three representative definitions. Sign in as a low-privilege test user and search for something you know exists. Confirm that only the intended search instance holds priority one. Confirm the old engine is decommissioned rather than merely ignored.
|
|
Add permanently to your PeopleTools patch test matrix
|
Global Search returns results for a non-administrator. Document-level security still filters correctly. Incremental indexing completes inside its window.
|
Those three lines in the last row take about two minutes to run, and they would have caught most of the cases in this article before the users did.
Final thoughts
Search is the feature nobody praises and everybody notices. When it works it is invisible. When it returns nothing it is the first item in the stand-up.
The short version of everything above: a green index proves that documents were written. It proves nothing at all about whether anyone is allowed to read them back. So establish the document count first, then start at the callback and work outwards.
And when you do find it, the index will have been fine the whole time. It usually is.
If you have hit this and the cause was something not on this list, please add it. The next person searching for indexing succeeded but no results will be very glad you did.
Reference
PeopleSoft PeopleTools 8.62, Search Technology: Working With Search Instances, for the callback properties, the Update deployed definitions link and namespace aliasing.
PeopleSoft PeopleTools 8.62, Search Technology: Administering Search Definitions and Search Categories, for Report Sync Issues, Reset Definition for Deploy and the Audit process.
PeopleSoft PeopleTools 8.62, Search Technology: Working with Authentication and Authorization, for source-level and document-level security.
PeopleSoft PeopleTools 8.62, Search Technology: Configuring SSL between PeopleSoft and Search Engine.
PeopleSoft PeopleTools 8.62, Search Technology: Troubleshooting OpenSearch Configuration, for common errors and log locations.
PeopleSoft PeopleTools 8.63, Search Technology, for the current release.
HEUG Technical community thread: 8.62 Global Search Indexing Runs but No Results, which prompted this guide. Thanks to the members who worked through it in public.
Gaurav Ohri, DBA Senior, The University of Texas System Office