diff --git a/quickstarts/analyze-data/geojson-to-vantage.md b/quickstarts/analyze-data/geojson-to-vantage.md index 6e01857e22..f25bb82a64 100644 --- a/quickstarts/analyze-data/geojson-to-vantage.md +++ b/quickstarts/analyze-data/geojson-to-vantage.md @@ -1,41 +1,41 @@ --- sidebar_position: 14 author: Rémi Turpaud -email: remi.turpaud@teradata.com +email: remi.turpaud@teradata.com, developer.relations@teradata.com page_last_update: February 14th, 2022 -description: How to load and use GeoJson documents with Teradata Vantage. -keywords: [geospatial, geojson, teradata, vantage, cloud data platform, analytics, maps, 4d analytics, open data] +description: How to load and use GeoJson documents with Teradata. +keywords: [geospatial, geojson, teradata, cloud data platform, analytics, maps, 4d analytics, open data] --- import TrialDocsNote from '../_partials/teradata_trial.mdx' import CommunityLinkPartial from '../_partials/community_link.mdx'; -# Use geographic reference data with Vantage +# Use geographic reference data with Teradata ## Overview -This post demonstrates how you can leverage any geographic dataset in GeoJson format and use it for geospatial analytics in Teradata Vantage, with just a few lines of code. +This post demonstrates how you can leverage any geographic dataset in GeoJson format and use it for geospatial analytics in Teradata, with just a few lines of code. Today we be gathering reference geographical data (official maps, points of interest, etc...) form public sources and use it to support our day to day analytics. -You will learn two methods to get your GeoJson data into Teradata Vantage: +You will learn two methods to get your GeoJson data into Teradata: 1. Load it as a single document and use native ClearScape analytics functions to parse it into a table usable for analytics. -2. Lightly transform it in native Python as we load it into Vantage to produce an analytics ready dataset. +2. Lightly transform it in native Python as we load it into Teradata to produce an analytics ready dataset. -The first method is a straig forward ELT pattern for semi-structured format processing in Vantage with a single SQL statement, the second one involves some lightweight preparation in (pure) Python and may allow more flexibility (for example to add early quality checks or optimize the load of large documents). +The first method is a straig forward ELT pattern for semi-structured format processing in Teradata with a single SQL statement, the second one involves some lightweight preparation in (pure) Python and may allow more flexibility (for example to add early quality checks or optimize the load of large documents). ## Prerequisites You will need: -- Access to a Teradata Vantage instance. +- Access to a Teradata Teradata instance. - A Python 3 interpreter - A SQL Client -## Option 1: Load a GeoJson document into Vantage -Here we will load a GeoJson document as a single Character Large OBject (CLOB) into the Vantage Data Store and use a single SQL statement, backed by ClearScape Analytics native functions, to parse this document into a usable structure for geospatial analytics. +## Option 1: Load a GeoJson document into Teradata +Here we will load a GeoJson document as a single Character Large OBject (CLOB) into the Teradata Data Store and use a single SQL statement, backed by ClearScape Analytics native functions, to parse this document into a usable structure for geospatial analytics. ### Get and load the GeoJson document The http://geojson.xyz/ website is a fantastic source for open geographical data in GeoJson format. We will load the "Populated Places" dataset that provides with a list of over 1000 significant world cities. @@ -54,20 +54,20 @@ with open(world_cities) as geo_json: jmap = jmap = geo_json.read() ``` -### Load the GeoJson document in Vantage +### Load the GeoJson document in Teradata -Modify this code as needed with your Vantage host name, user name and specify an advanced login mechanism if any (eg. LDAP, Kerberos...) with the _logmech_ parameter if you need to. +Modify this code as needed with your Teradata host name, user name and specify an advanced login mechanism if any (eg. LDAP, Kerberos...) with the _logmech_ parameter if you need to. All the connection parameters are documented on the teradatasql PyPi page there: https://pypi.org/project/teradatasql/ -The code below simply creates a Vantage connection, and opens a cursor creating a table and loading it with our file. +The code below simply creates a Teradata connection, and opens a cursor creating a table and loading it with our file. ```python import teradatasql import getpass -tdhost='' -tdUser='' +tdhost='' +tdUser='' -# Create a connection to Teradata Vantage +# Create a connection to Teradata con = teradatasql.connect(None, host=tdhost, user=tdUser, password=getpass.getpass()) # Create a table named geojson_src and load the JSON map into it as a single CLOB @@ -76,12 +76,12 @@ with con.cursor () as cur: r=cur.execute ("insert into geojson_src (?, ?)", ['cities',jmap]) ``` -### Use the map from Vantage +### Use the map from Teradata -Now open your favourite **SQL client** and connect to your Vantage system. +Now open your favourite **SQL client** and connect to your Teradata system. We will use ClearScape analytics JSON functions to parse our GeoJson document and extract the most relevant properties and the geometry itself (the coordinates of the city) for each feature (each feature representing a city in this example). -We then use the GeomFromGeoJSON function to cast our geometry as a native Vantage geometry data type (ST_GEOMETRY). +We then use the GeomFromGeoJSON function to cast our geometry as a native Teradata geometry data type (ST_GEOMETRY). For user convenience, will wrap all this SQL code in a view: @@ -146,11 +146,11 @@ Result: |1.9265006861079421e+06| -## Option 2: Prepare a GeoJson document with Python and load it into Vantage +## Option 2: Prepare a GeoJson document with Python and load it into Teradata -The previous example demonstrated how to load a complete document as a large object into Teradata Vantage and use built in analytic functions to parse it into a usable dataset. +The previous example demonstrated how to load a complete document as a large object into Teradata and use built in analytic functions to parse it into a usable dataset. -This is convenient but limited: we need to parse this document every time we need to use it, as the original document is not directly usable for analytics, JSON documents are currently limited to 16MB in Vantage and it may be inconvenient to fix data quality or formatting issues within the document stored as a CLOB. +This is convenient but limited: we need to parse this document every time we need to use it, as the original document is not directly usable for analytics, JSON documents are currently limited to 16MB in Teradata and it may be inconvenient to fix data quality or formatting issues within the document stored as a CLOB. In this example, we will parse our JSON document using the Python json package and load it as a table that can be used directly and efficiently for analysis. @@ -195,7 +195,7 @@ print(countries_json['features'][0]['geometry']['coordinates']) What we have here is a collection of GeoFeatures (as earlier). -We will now lightly model this data in a Vantage table, for that: +We will now lightly model this data in a Teradata table, for that: - We will load each feature as a raw. - We will extract the properties that look interesting for immediate analysis (in our example, the country name and ISO code). @@ -213,21 +213,21 @@ For example: NB: Not featured here, but recommended for richer datasets, consider loading the entire and original feature payload as a separate column (this is a JSON document). This will allow you to go back to the original record and extract new properties that you may have missed during your first analysis but have become relevant, directly in SQL and without having to reload the file entirely. -### Create a Vantage connection and load our file in a staging table +### Create a Teradata connection and load our file in a staging table -Modify this code as needed with your Vantage host name, user name and specify an advanced login mechanism if any (eg. LDAP, Kerberos...) with the _logmech_ parameter if you need to. +Modify this code as needed with your Teradata host name, user name and specify an advanced login mechanism if any (eg. LDAP, Kerberos...) with the _logmech_ parameter if you need to. All the connection parameters are documented on the teradatasql PyPi page there: https://pypi.org/project/teradatasql/ -The code below simply creates a Vantage connection, and opens a cursor creating a table and loading it with our list. +The code below simply creates a Teradata connection, and opens a cursor creating a table and loading it with our list. ```python import teradatasql import getpass -tdhost='' -tdUser='' +tdhost='' +tdUser='' -# Create a connection to Teradata Vantage +# Create a connection to Teradata con = teradatasql.connect(None, host=tdhost, user=tdUser, password=tdPassword) # Create a table and load our country names, codes, and geometries. @@ -240,7 +240,7 @@ with con.cursor () as cur: The code below performs the table creation from the Python interpreter, you can also run the _sql_ statement defined below in your prefered SQL client you might as well simply define this logic as a SQL view to avoid having to refresh this table. -We will use ClearScape analytics the GeomFromGeoJSON function to cast our geometry as a native Vantage geometry data type (ST_GEOMETRY). +We will use ClearScape analytics the GeomFromGeoJSON function to cast our geometry as a native Teradata geometry data type (ST_GEOMETRY). ```python @@ -288,6 +288,4 @@ Note that none of the above code does not implement any control procedure or che Consider using [SQLAlchemy ORM](https://pypi.org/project/teradatasqlalchemy/) if you are defining your pipeline in Python, xref:dbt.adoc[dbt], or your favorite ELT and orchestration toolset to create your products you can operationalize. -You now can know how to get any open geographic dataset and use it to augment your analytics with Teradata Vantage! - - \ No newline at end of file +You now can know how to get any open geographic dataset and use it to augment your analytics with Teradata! \ No newline at end of file