Refreshing Dremio Assets
Dremio refreshes datasets in the interval we setup at the data source level. There are two sections for metadata refresh. Dataset Discovery: Refresh interval for top-level source object names such as names of DBs and tables. This is a lightweight operation. Dataset Details: Metadata Dremio needs for query planning such as information on fields, types, shards, statistics and locality. Any new data ingested into the source should be available instantaneously. However, there can be some changes in the DDL based on use cases. On that case, until the Dataset Details are refreshed by Dremio, the new records will not appear. Refreshing this setting frequently can be a overhead. So, in case of any particular datasets need to be refreshed frequently better we can refresh those on-demand rather than making the whole data source busier. Dremio provides few ways to refresh datasets (physical data sets aka PDS). Let's see. REST API Fetch the Entity Id pds = 'Hive.demoschema.demotable'