[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Need help on mismatch of returned data

Title: Default Disclaimer Daimler AG
I tried adding a query with a bbox, during and equals filter to our existing unit test, and it returned as expected (failing the test in this case, as it expects 10 features and only 1 was returned):

"dtg DURING 2017-06-05T04:03:00.0000Z/2017-06-07T04:04:00.0000Z and bbox(geom, 5, 5, 15, 15) AND name = 'test1'"

[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.631 s <<< FAILURE! - in org.locationtech.geomesa.fs.FileSystemDataStoreTest
[ERROR] FileSystemDataStore should::support transforms(org.locationtech.geomesa.fs.FileSystemDataStoreTest)  Time elapsed: 0.025 s  <<< FAILURE!
org.specs2.reporter.SpecFailureAssertionFailedError:
There are 2 failures
'List(ScalaSimpleFeature:1:test1|101|Tue Jun 06 04:03:02 UTC 2017|POINT (10 10.1))' doesn't have length 10 but length 1
'List(ScalaSimpleFeature:1:test1|101|Tue Jun 06 04:03:02 UTC 2017|POINT (10 10.1))' doesn't have length 10 but length 1

https://github.com/locationtech/geomesa/blob/master/geomesa-fs/geomesa-fs-datastore/src/test/scala/org/locationtech/geomesa/fs/FileSystemDataStoreTest.scala#L177

The test uses a FeatureReader.

Thanks,

Emilio

On 5/13/19 8:34 AM, Emilio Lahr-Vivaz wrote:
Hello,

Are you able to get any results back at all from your programmatic query? If the filter is working through the CLI tools, then I would tend to think that the issue is due to a different classpath or environment setup in your programmatic environment. As an additional step, you can modify the logging configuration in the CLI tools under conf/log4j.properties and see what the FilterCompat log shows there. Another thing to try is to use a FeatureSource instead of a FeatureReader - they should be equivalent, but the export command uses a FeatureSource, so maybe that is the difference: https://github.com/locationtech/geomesa/blob/master/geomesa-tools/src/main/scala/org/locationtech/geomesa/tools/export/ExportCommand.scala#L106

Thanks,

Emilio

On 5/13/19 2:23 AM, christian.sickert@xxxxxxxxxxx wrote:

To add one more piece of information, this is what gets logged when we programmatically query the data using the GeoTools API:

 

INFO  org.apache.parquet.filter2.compat.FilterCompat - Filtering using predicate: and(and(and(and(and(gteq(geom.x, -180.0), gteq(geom.y, -90.0)), lteq(geom.x, 180.0)), lteq(geom.y, 90.0)), and(gteq(dtg, 1555198800000), lteq(dtg, 1555203600000))), eq(vin, Binary{"VINSYSTEMTEST1"}))

 

Hope that helps.

 

Von: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> Im Auftrag von christian.sickert@xxxxxxxxxxx
Gesendet: Montag, 13. Mai 2019 08:20
An: geomesa-users@xxxxxxxxxxxxxxxx
Betreff: Re: [geomesa-users] Need help on mismatch of returned data

 

Hi Emilio,

 

thanks for your hint. However that doesn’t change the behaviour.

 

Programmatically querying our data using the following filter constraints

-          Bounding box: -180,90,180,90 (the whole world)

-          2019-04-13T23:40:00Z <= dtg <= 2019-04-14T01:00Z

-          vin = VINSYSTEMTEST1

and the code I’ve sent to you with my last message doesn’t return any results (although there should be exactly one).

 

From debugging the program code I’ve extracted the following CQL filter for the :

dtg BETWEEN 2019-04-13T23:40:00+00:00 AND 2019-04-14T01:00:00+00:00 AND BBOX(geom, -180.0,-90.0,180.0,90.0) AND vin = 'VINSYSTEMTEST1'

 

Using that very same cql-filter with the command line client “geomesa-fs export” on the test data does return the intended result (exactly one hit).

 

Could you please have a look at it? Or should we address this to the geotools maintainers?

 

Thanks,

Christian

 

 

Von: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> Im Auftrag von Emilio Lahr-Vivaz
Gesendet: Freitag, 10. Mai 2019 17:11
An: geomesa-users@xxxxxxxxxxxxxxxx
Betreff: Re: [geomesa-users] Need help on mismatch of returned data

 

Hello,

I notice that in your code, you are using an 'equals' query on VIN, while on the command-line you are using a 'like' with wildcards. Could you try using a consistent predicate between the two? I'm not entirely sure that 'like' is handled correctly in the FSDS, tbh.

Thanks,

Emilio

On 5/10/19 3:59 AM, christian.sickert@xxxxxxxxxxx wrote:

Hi Emilio,

 

This is Christian, a colleague of Markus - the initial requester, who is currently out-of-office. Maybe I can answer your question.

 

We’re using the GeoTools API to query the data. We basically do it in the same way as described in the geomesa-tutorial https://github.com/geomesa/geomesa-tutorials/blob/master/geomesa-tutorials-common/src/main/java/org/geomesa/example/transformations/GeoMesaQueryTutorial.java

 

First, we build an org.geotools.data.Query instance using org.opengis.filter.FilterFactory2. We’re applying temporal and spatial constraints as well as a constraint on a custom attribute called VIN:

 

Query query = new Query(typeName);
List<Filter> filters = new ArrayList<>();

FilterFactory2 filterFactory = CommonFactoryFinder.getFilterFactory2();

// temporal filter:
filters.add(filterFactory.between(
      filterFactory.property(EventSimpleFeature.DATE_TIME_GROUP),
      filterFactory.literal(dateFrom),
      filterFactory.literal(dateTo)));

// spatial filter:
filters.add(filterFactory
      .bbox(filterFactory.property(EventSimpleFeature.GEOMETRY),
            boundingBox.getMinX(), boundingBox.getMinY(),
            boundingBox.getMaxX(), boundingBox.getMaxY(), "EPSG:4326"));

// vin filter:
filters.add(filterFactory
      .equals(filterFactory.property(EventSimpleFeature.VIN), filterFactory.literal(vin)));

query.setFilter(filterFactory.and(filters));
query.setPropertyNames(fieldNames);
return query;

 

With that query we then instantiate an org.geotools.data.FeatureReader to read the filtered (according the given query) entries:

 

FeatureReader<SimpleFeatureType, SimpleFeature> reader = dataStore.getFeatureReader(query, Transaction.AUTO_COMMIT);
while (reader.hasNext()) {
   SimpleFeature feature = reader.next();

   // do something with the feature ...
}

 

Is that sufficient for a further analysis on your side?

 

Thanks for your support!

 

Best regards

Christian

 

 

Von: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> Im Auftrag von Emilio Lahr-Vivaz
Gesendet: Mittwoch, 8. Mai 2019 22:32
An: geomesa-users@xxxxxxxxxxxxxxxx
Betreff: Re: [geomesa-users] Need help on mismatch of returned data

 

Hello,

Could you provide the code you're using to query programmatically?

Thanks,

Emilio

On 5/8/19 10:37 AM, bosch.thamm@xxxxxxxxxxxxxx wrote:

Hello,

 

In a current project we are using geomesa-fs-data-store for our network-measurements. We are collecting these network-measurements from driving vehicles.

 

The issues is that we are facing a mismatch with returned data between “geomesa-fs export” and querying geomesa programmatically.

This is the geomesa-fs export-command: ./geomesa-fs export -p ………./geomesa-data-store/ -f ModemState --cql "dtg>=2019-04-13T23:40:00Z AND dtg<=2019-04-14T01:00:00Z AND vin LIKE '%VINSYSTEMTEST1%'"

 

This is the query from a debug-statement when querying geomesa programmatically:

2019-05-08 16:16:16.073 [http-nio-8130-exec-2] INFO  org.apache.parquet.filter2.compat.FilterCompat - Filtering using predicate: and(and(and(and(and(gteq(geom.x, -180.0), gteq(geom.y, -90.0)), lteq(geom.x, 180.0)), lteq(geom.y, 90.0)), and(gteq(dtg, 1555198800000), lteq(dtg, 1555203600000))), eq(vin, Binary{"VINSYSTEMTEST1"}))

 

While the export returns 1 event, the programmatic query returns 0 events.

The event is found in the programmatic case when extending the search-interval to an earlier time. E.g. from 2019-04-13T23:40:00Z to 2019-04-13T00:40:00Z

 

I am also attaching the test-data.

 

We are using:

Geomesa-fs-datastore_2.11:2.2.0

Geomesa-fs-storage-parquet_2.11:2.2.0

Partitioning-Scheme: daily, z2-2bit

 

The storage-type is parquet on AWS S3.

 

I hope you can help. Thanks,

Markus Thamm


If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

 


If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.


 

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

 


If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.

 

Default Disclaimer Daimler AG
If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users