PS Query & SQL

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

Here is one method which I have used in multiple queries.... This "delta" approach requires that the query is scheduled and meets various criteria to ensure it is looking at the desired "last run"....

As with plenty of PeopleSoft, "it depends"... And there can be other approaches which could vary based upon what data you are looking for / what is known.

Have fun.

David Ehrlich | Senior Business System Analyst - Admissions, Student Records, Data & Reporting

Duke University | Student Information Services & System (SISS)

david.ehrlich@duke.edu | 919-684-1270

www.sissoffice.duke.edu

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

Depending on what you are doing with the final dataset, you could consider using Opensearch/Insights. We have a few running, all with PSQUERY as the datasource. These run as full datasets overnight, then every half hour during the day using the Incremental method.

Some of the datasets just feed into the Global Search on the page header. e.g.

Some feed into Insights (Kibana) reports/dashboards for users to further drill down into data.

Maybe worth looking at. I think 8.62 has further enhancements as to how Incremental PSQUERYs work. In these cases though, although PSQUERY is running an Incremental load, the results are added to the main dataset. So you would need to set a Status field or something to identify the latest additions. 

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

Depending on what you are doing with the final dataset, you could consider using Opensearch/Insights. We have a few running, all with PSQUERY as the datasource. These run as full datasets overnight, then every half hour during the day using the Incremental method.

Some of the datasets just feed into the Global Search on the page header. e.g.

Some feed into Insights (Kibana) reports/dashboards for users to further drill down into data.

Maybe worth looking at. I think 8.62 has further enhancements as to how Incremental PSQUERYs work. In these cases though, although PSQUERY is running an Incremental load, the results are added to the main dataset. So you would need to set a Status field or something to identify the latest additions. 

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

Hi Mital,

I can show the basics of Insights/OpenSearch via a video using 8.60.10 if that is helpful. But, do you use the Global Search Bar at the top of the page, and have an OpenSearch server running (previously this was all called Elastic Search and Kibana).

> Peopletools > Search Framework>Administration> Define Search Instance

They would be needed.

Thanks

Daron

Depending on what you are doing with the final dataset, you could consider using Opensearch/Insights. We have a few running, all with PSQUERY as the datasource. These run as full datasets overnight, then every half hour during the day using the Incremental method.

Some of the datasets just feed into the Global Search on the page header. e.g.

Some feed into Insights (Kibana) reports/dashboards for users to further drill down into data.

Maybe worth looking at. I think 8.62 has further enhancements as to how Incremental PSQUERYs work. In these cases though, although PSQUERY is running an Incremental load, the results are added to the main dataset. So you would need to set a Status field or something to identify the latest additions. 

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

If I only need the date field, I just use an expression, change the type to "Date" and then put the date/time field in there and then "Use Field".  

Hi Mital,

I can show the basics of Insights/OpenSearch via a video using 8.60.10 if that is helpful. But, do you use the Global Search Bar at the top of the page, and have an OpenSearch server running (previously this was all called Elastic Search and Kibana).

> Peopletools > Search Framework>Administration> Define Search Instance

They would be needed.

Thanks

Daron

Depending on what you are doing with the final dataset, you could consider using Opensearch/Insights. We have a few running, all with PSQUERY as the datasource. These run as full datasets overnight, then every half hour during the day using the Incremental method.

Some of the datasets just feed into the Global Search on the page header. e.g.

Some feed into Insights (Kibana) reports/dashboards for users to further drill down into data.

Maybe worth looking at. I think 8.62 has further enhancements as to how Incremental PSQUERYs work. In these cases though, although PSQUERY is running an Incremental load, the results are added to the main dataset. So you would need to set a Status field or something to identify the latest additions. 

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You

All Query Experts out there,

Have you ever worked on a Query, Connected Query,Composite Query request where you can send incremental data (processing only the new or changed data since the last processing cycle, rather than reprocessing the entire dataset?

Thank You