High Water Mark - suggestions for Incremental extraction of Sales Invoice Data (lastModified does not work)
Hi
I am looking to incrementally load transactions from
{{company_file_uri}}/{{company_file_id}}/Sale/Invoice/Item?$filter=Date ge datetime'{{HWMvalue}}$orderby=LastModified
the High water Mark - {{HWMValue}} maximum value of LastModified is retrieved from the already extracted data -
Having some challenges.
- A fair chunk of the available data has a LastModified date of 1970-01-01 . I assume this is a placeholder for "null" this means I cant use lastModified to fetch conservative amounts of data from the API
I have attempted to use the sales invoice "Date" field as a high water mark (While this Date not ideal to use for loading incrementally at least , hopefully, it should allow me to load in historical data) - there are quite a few rows I'm expecting - 35099 Headers and 497875 Lines.
Is there a best practice strategy for
1. Bulk fetching historical data
2. Incremental fetching Sales Invoices