Forum Discussion

Phocas's avatar
Phocas
Contributing User
3 years ago

High Water Mark - suggestions for Incremental extraction of Sales Invoice Data (lastModified does not work)

Hi 

I am looking to incrementally load transactions from 

{{company_file_uri}}/{{company_file_id}}/Sale/Invoice/Item?$filter=Date ge datetime'{{HWMvalue}}$orderby=LastModified

 

the High water Mark - {{HWMValue}} maximum value of LastModified is retrieved from the already extracted data - 

Having some challenges. 

- A fair chunk of the available data has a LastModified date of 1970-01-01 . I assume this is a placeholder for "null" this means I cant use lastModified to fetch conservative amounts of data from the API 

 

I have attempted to use the sales invoice "Date" field as a high water mark (While this Date not ideal to use for loading incrementally at least , hopefully,  it should allow me to load in historical data) - there are quite a few rows I'm expecting -   35099 Headers and 497875 Lines.

Is there a best practice strategy for

1. Bulk fetching historical data 

2. Incremental fetching Sales Invoices  

 

  • Hi Phocas 

     

    Just in relation to the date 1 Jan 1970, this is the default Last Modified date for all transactions that were in the file at time of upgrade from v19. Anything entered in Ar2021 should have a specific date.

     

    Regards

    Gavin