Transferring data from server to s3 buckets

[mccabete]

@robertdfield has been transferring data from a server at GISS into the veda-data buckets. The way he was doing this was built around how the old ADE handled ssh-ing into workspaces, and the authentication for that using the AWS client broke down with the transfer to the new ADE.

We are curious if there is a more official way of building a pipeline for transferring data (in this case CSVs) into the veda-data s3 buckets and updating that data regularly. @wildintellect this may be a VEDA question, not a MAAP question. And @ashiklom -- do you happen to remember how the authentication worked when you and Robert set it up originally?

Thanks for the help!

[ranchodeluxe]

There are many potential scheduled job workflows we can use to make this happen whether they are in version control, a new ADE or AWS. I think the questions that needs to be answered here are about GISS.

• Can the CSVs on their server be made public?

• If not able to make public do we have the ability to put them on an HTTP Server, FTP, version control (GH or Gitlab) or even consumer-oriented cloud (Dropbox etc) where some workflow has long-term credentials to access them?

[robertdfield]

Hi!

If by public you mean access to the GISS server, that's probably tricky given that it's on the NASA network and only accessible with AUID login.

However, the data Tess is referring to is now publicly available on the NCCS Dataportal:
https://portal.nccs.nasa.gov/datashare/GlobalFWI/StnFWI/19900101.NRT/

This is fire weather data at individual weather stations. I can explain more, but the specific data to first get on AWS would be:

1. This list of weather stations:
   https://portal.nccs.nasa.gov/datashare/GlobalFWI/StnFWI/19900101.NRT/ManualStationList.fwiStnList.minHourlyCompleteness.10.ISDGHCNMaxPrecDiff.NaN.1990.2023.csv

2. The fire weather data at those weather stations, which are in the Hourly.csv and Daily.csv files in the 'FWI' subdirectory:
   https://portal.nccs.nasa.gov/datashare/GlobalFWI/StnFWI/19900101.NRT/FWI/

Does this seem workable? The production isn't on a cron job yet, but I update it regularly. You'll also see compressed files for each station if those are useful.

Thank you!
Robert

[wildintellect]

If the data is now on those public urls, can the code be updated to use them directly from that location?

[mccabete]

@robertdfield - is the data only recently public? Was there a reason we were moving it to s3 buckets beyond access (EIS reasons?)?

[robertdfield]

It's been public since sometime in August. When my AWS setup worked, I think that was the cleanest way for internal use, but happy to cut that part out if the Dataportal is an option.

[mccabete]

@robertdfield it could be possible to go directly from your now-public Dataportal, but it will be a fair amount of rebuilding on my part. We probably need to talk about differences in data-organization structure, and how it would be prioritized against other stuff.

[mccabete]

@smohiudd This is the same data that is now the new fwi collection in the API. For front-end applications, could we be reading directly from the Dataportal?

[smohiudd]

Unfortunately not the way it is set up. We would need the data served in geospatial format (i.e. geojson) so it can be visualized easily in a front end environment. However if we're talking about a jupyter notebook then importing the CSV as is will work. One thing to note is that the way the CSVs are currently generated, we need to do a table join to get the lat/lon columns. So there is some transformation of the raw data needed before it's ready for visualization.

[mccabete]

@robertdfield The data joins to get lat and lon are also what would take me time re-building everything when transferring to a new data source, like @smohiudd. Could this be something that you automatically export?