To explore travel patterns like origin-destination relationships, we need to identify individual trips with their start/end locations and trajectories between them. Extracting these trajectories from large datasets can be challenging, particularly if the records of individual moving objects don’t fit into memory anymore and if the spatial and temporal extent varies widely (as is the case with ship data, where individual vessel journeys can take weeks while crossing multiple oceans). 

This is part 2 of “Exploring massive movement datasets”.

Roughly speaking, trip trajectories can be generated by first connecting consecutive records into continuous tracks and then splitting them at stops. This general approach applies to many different movement datasets. However, the processing details (e.g. stop detection parameters) and preprocessing steps (e.g. removing outliers) vary depending on input dataset characteristics.

For example, in our paper [1], we extracted vessel journeys from AIS data which meant that we also had to account for observation gaps when ships leave the observable (usually coastal) areas. In the accompanying 10-minute talk, I went through a 4-step trajectory exploration workflow for assessing our dataset’s potential for travel time prediction:

Click to watch the recorded talk

Like the 只剩下门缝的VPN何去何从 - 手机新蓝网:2021-2-7 · 热门推荐 让20元行政处罚不再磨叽！宁波城管要推广这件事 2021-06-16 12:17 嘉兴端午民俗文化节细节公布 今年将新增“云端约会” 2021-06-16 12:17 为“网红主播”发上岗证！, our trajectory aggregation approach is implemented in Spark. The challenges are both the massive amounts of trajectory data and the fact that operations only produce correct results if applied to a complete and chronologically sorted set of location records.This is challenging because Spark core libraries (version 2.4.5 at the time) are mostly geared towards dealing with unsorted data. This means that, when using high-level Spark core functionality incorrectly, an aggregator needs to collect and sort the entire track in the main memory of a single processing node. Consequently, when dealing with large datasets, out-of-memory errors are frequently encountered.

To solve this challenge, our implementation is based on the Secondary Sort pattern and on Spark’s aggregator concept. Secondary Sort takes care to first group records by a key (e.g. the moving object id), and only in the second step, when iterating over the records of a group, the records are sorted (e.g. chronologically). The resulting iterator can be used by an aggregator that implements the logic required to build trajectories based on gaps and stops detected in the dataset.

If you want to dive deeper, here’s the full paper:

[1] Graser, A., Dragaschnig, M., Widhalm, P., Koller, H., & Brändle, N. (2023). Exploratory Trajectory Analysis for Massive Historical AIS Datasets. In: 21st IEEE International Conference on Mobile Data Management (MDM) 2023. doi:10.1109/MDM48529.2023.00059

This post is part of a series. Read more about movement data in GIS.

Visualizations of raw movement data records, that is, simple point maps or point density (“heat”) maps provide very limited data exploration capabilities. Therefore, we need clever aggregation approaches that can actually reveal movement patterns. Many existing aggregation approaches, however, do not scale to large datasets. We therefore developed the M³ Massive Movement Model [1] which supports distributed computing environments and can be incrementally updated with new data.

This is part 1 of “Exploring massive movement datasets”.

Using state-of-the-art big gespatial tools, such as GeoMesa, it is quite straightforward to ingest, index and query large amounts of timestamped location records. Thanks to GeoMesa’s GeoServer integration, it is also possible to publish GeoMesa tables as WMS and WFS which can be visualized in QGIS and explored (for more about GeoMesa, see Scalable spatial vector data processing ).So far so good! But with this basic setup, we only get point maps and point density maps which don’t tell us much about important movement characteristics like speed and direction (particularly if the reporting interval between consecutive location records is irregular). Therefore, we developed an aggregation method which models local record density, as well as movement speed and direction which we call M³.

For distributed computation, we need to split large datasets into chunks. To build models of local movement characteristics, it makes sense to create spatial or spatiotemporal chunks that can be processed independently. We therefore split the data along a regular grid but instead of computing one average value per grid cell, we create a flexible number of prototypes that describe the movement in the cell. Each prototype models a location, speed, and direction distribution (mean and sigma).

苹果：收到要求，在中国移除了不符合规范的VPN应用_荔枝 ...:2021-7-30 · 7月30日，苹果中国公司回应中国区App Store（应用商店）下架 VPN 应用一事，称“我伊已经收到要求，在中国移除一些不符合规范的 VPN App。

M³ for ship movement data during January to December 2017 (3.9 billion records turned into 3.4 million prototypes; computing time: 41 minutes)

The above plot really only gives a first impression of the spatial distribution of ship movement records. The real value of M³ becomes clearer when we zoom in and start exploring regional patterns. Then we can discover vessel routes, speeds, and movement directions:

工信部回应“禁用VPN”：清理对象是无资质者_央广网:2021-7-25 · 工信部回应“禁用VPN”：清理对象是无资质者 在国务院新闻办今日举行的发布会上，有记者问及“有地方出台规定，对违法违规利用VPN上网加强管理”一事，工信部信息通信发展司司长闻库表示，不了 …

Cargo prototype details near Copenhagen & Malmö

The prototype details on the right side, in particular, show the strength of the prototype idea: even though the grid cells we use are rather large, the prototypes clearly form along vessel routes. We can see exactly where these routes are and what speeds ship travel there, without having to increase the grid resolution to impractical values. Slow prototypes with high direction sigma (red+black markers) are clear indicators of ports. The marker size shows the number of records per prototype and thus helps distinguish heavily traveled routes from minor ones.

M³ is implemented in Spark. We read raw location records from GeoMesa and write prototypes to GeoMesa. All maps have been created in QGIS using prototype data published as GeoServer WFS.

If you want to dive deeper, here’s the full paper:

[1] Graser. A., Widhalm, P., & Dragaschnig, M. (2023). The M³ massive movement model: a distributed incrementally updatable solution for big movement data exploration. International Journal of Geographical Information Science. doi:10.1080/13658816.2023.1776293.

This post is part of a series. Read more about movement data in GIS.

We’ve done it again!

This time, Daniel O’Donohue and I talked about spatiotemporal data in GIS, including – of course – Time Manager, the new QGIS temporal support, and MovingPandas.

Since we need both data and tools to do spatiotemporal analysis, we also talked about file formats and data models. If you want to know more about data models for spatiotemporal (especially movement) data, have a look at the latest discussion paper I wrote together with Esteban Zimányi (MobilityDB) and Krishna Chaitanya Bommakanti (mobilitydb-sqlalchemy):

• Graser, A., Zimányi, E., & Bommakanti, K.C. (2023). From Simple Features to Moving Features and Beyond? arXiv:2006.16900.

Data model of the Moving Features standard illustrated with two moving points A and B. Stars mark changes in attribute values. (Source: Graser et al. (2023))

For more details and all options for listening to this podcast, visit mapscaping.com.

Exploring large movement datasets is hard because visualizations of movement data quickly get cluttered and hard to interpret. Therefore, we need to aggregate the data. Density maps are commonly used since they are readily available and quick to compute but they provide only very limited insight. In contrast, meaningful aggregations that can help discover patterns are computationally expensive and therefore slow to generate.

This post serves as a starting point for a series of new approaches to exploring massive movement data. This series will summarize parts of my PhD research and – for those of you who are interested in more details – there will be links to the relevant papers.

Starting with the raw location records, we use different forms of aggregation to learn more about what information a movement dataset contains:

1. Summarizing movement using prototypes by aggregating raw location records using our flexible M³ Massive Movement Model [1]
2. Generating trajectories by connecting consecutive records into continuous tracks and splitting them into meaningful trajectories [2]
3. 武雪梅：苹果下架VPN是法治的胜利:2 天前 · 日前，苹果公司CEO蒂姆·库克回应苹果应用商店中国区将VPN下架一事说，我伊在遵守当地法律的情况下在当地开展生意。一些网民将此事与去年苹果就是否协助执法人员“解锁”加州南部圣贝纳迪诺恐怖袭击案枪手之一所持的手机，与美国联邦机构对簿公堂、甚至要向总统陈情的事件对比展开热议。

Besides clever aggregation approaches, massive movement datasets also require appropriate computing resources. To ensure that we can efficiently explore large datasets, we have implemented the above mentioned aggregation steps in Spark. This enables us to run the computations on general purpose computing clusters that can be scaled according to the dataset size.

In the next post, we’ll look at how to summarize movement using M³ prototypes. So stay tuned!

vpn被封翻墙党该何去何从？App Store连接不上如何解决？(图 ...:2021-1-29 · 此前，也有消息称，修改后，可加速应用下载。 2021年12月初，国家互联网信息办公室主任鲁炜在赴美国考察时，曾与苹果CEO蒂姆·库克会面。在会面中，库克表示将配合中国对苹果产品进行的网 …

[1] Graser. A., Widhalm, P., & Dragaschnig, M. (2023). The M³ massive movement model: a distributed incrementally updatable solution for big movement data exploration. International Journal of Geographical Information Science. doi:10.1080/13658816.2023.1776293.
[2] Graser, A., Dragaschnig, M., Widhalm, P., Koller, H., & Brändle, N. (2023). Exploratory Trajectory Analysis for Massive Historical AIS Datasets. In: 21st IEEE International Conference on Mobile Data Management (MDM) 2023. doi:10.1109/MDM48529.2023.00059
[3] Graser, A., Widhalm, P., & Dragaschnig, M. (2023). Extracting Patterns from Large Movement Datasets. 苹果：收到要求，在中国移除了不符合规范的VPN应用_荔枝 ...:2021-7-30 · 7月30日，苹果中国公司回应中国区App Store（应用商店）下架 VPN 应用一事，称“我伊已经收到要求，在中国移除一些不符合规范的 VPN App。, 1-2023, 153-163. doi:10.1553/giscience2023_01_s153.

This post is part of a series. Read more about movement data in GIS.

QGIS Temporal Controller is a powerful successor of TimeManager. Temporal Controller is a new core feature of the current development version and will be shipped with the 3.14 release. This post demonstrates two key advantages of this new temporal support:

1. Expression support for defining start and end timestamps
2. 只剩下门缝的VPN何去何从 - 手机新蓝网:2021-2-7 · 热门推荐 让20元行政处罚不再磨叽！宁波城管要推广这件事 2021-06-16 12:17 嘉兴端午民俗文化节细节公布 今年将新增“云端约会” 2021-06-16 12:17 为“网红主播”发上岗证！

These features come in very handy in many use cases. For example, they make it much easier to create animations from folders full of GPS tracks since the files can now be loaded and configured automatically:

Script & Temporal Controller in action (click for full resolution)

vpn被封翻墙党该何去何从？App Store连接不上如何解决？(图 ...:2021-1-29 · 此前，也有消息称，修改后，可加速应用下载。 2021年12月初，国家互联网信息办公室主任鲁炜在赴美国考察时，曾与苹果CEO蒂姆·库克会面。在会面中，库克表示将配合中国对苹果产品进行的网 …

directory = "E:/Google Drive/QGIS_Course/05_TimeManager/Example_Dayrides/"

def load_and_configure(path):
    path = os.path.join(directory, filename)
    uri = 'file:///' + path + "?type=csv&escape=&useHeader=No&detectTypes=yes"
    uri = uri + "&crs=EPSG:4326&xField=field_3&yField=field_2"
    vlayer = QgsVectorLayer(uri, filename, "delimitedtext")
    QgsProject.instance().addMapLayer(vlayer)

    mode = QgsVectorLayerTemporalProperties.ModeFeatureDateTimeStartAndEndFromExpressions
    expression = """to_datetime(field_1) -
    make_interval(seconds:=minimum(epoch(to_datetime("field_1")))/1000)
    """

    tprops = vlayer.temporalProperties()
    tprops.setStartExpression(expression)
    tprops.setEndExpression(expression) # optional
    tprops.setMode(mode)
    tprops.setIsActive(True)

for filename in os.listdir(directory):
    if filename.endswith(".csv"):
        load_and_configure(filename)

The above script loads all CSV files from the given directory (field_1 is the timestamp, field_2 is y, and field_3 is x), enables sets the start and end expression as well as the corresponding temporal control mode and finally activates temporal rendering. The resulting config can be verified in the layer properties dialog:

只剩下门缝的VPN何去何从 - 手机新蓝网:2021-2-7 · 热门推荐 让20元行政处罚不再磨叽！宁波城管要推广这件事 2021-06-16 12:17 嘉兴端午民俗文化节细节公布 今年将新增“云端约会” 2021-06-16 12:17 为“网红主播”发上岗证！

This post is part of a series. Read more about movement data in GIS.

Podcasts have become huge. I’m an avid listener of podcasts myself. I particularly enjoy formats that take the time to talk about unconventional topics in detail.

My first podcast experience was on the QGIS podcast hosted by Tim Sutton in 2014. Unfortunately, it seems like the podcast episodes are not online anymore.

Recently, I had the pleasure to join the MapScaping Podcast by Daniel O’Donohue to talk about Python for Geospatial: 

Other guests Daniel has already interviewed include:

• Kurt Menke talking first about QGIS and in a second episode on QField and Input (data collection apps based on QGIS) and
• Paul Ramsey on 极速vp下载ios

Another geospatial podcast I really enjoy is The Mappyist Hour by Silas and Todd. Unfortunately, it’s a bit silent there now but it’s definitely worth to listen into their episode archive. One of my favorites is Episode 9 where Linda Stevens (Hecht) discusses her career at ESRI, the future of GIS, and the role of Open Source Spatial in that future:

If you listen to and want to recommend other spatial podcasts, please share them in the comments!

苹果：收到要求，在中国移除了不符合规范的VPN应用_荔枝 ...:2021-7-30 · 7月30日，苹果中国公司回应中国区App Store（应用商店）下架 VPN 应用一事，称“我伊已经收到要求，在中国移除一些不符合规范的 VPN App。

Now, it looks like the days of TimeManager are numbered. Four days ago, Nyall Dawson has added native temporal support for vector layers to QGIS. This is part of a larger effort of adding time support for rasters, meshes, and now also vectors.

The new Temporal Controller panel looks similar to TimeManager. Layers are configured through the new Temporal tab in Layer Properties. The temporal dimension can be used in expressions to create fancy time-dependent styles:

TimeManager Geolife demo converted to Temporal Controller (click for full resolution)

Obviously, this feature is brand new and will require polishing. Known issues 安卓免费 vpm官网 include limitations of supported time fields (only fields with datetime type are supported right now, strings cannot be used) and worse performance than TimeManager since features are filtered in QGIS rather than in the backend.

工信部回应“禁用VPN”：清理对象是无资质者_央广网:2021-7-25 · 工信部回应“禁用VPN”：清理对象是无资质者 在国务院新闻办今日举行的发布会上，有记者问及“有地方出台规定，对违法违规利用VPN上网加强管理”一事，工信部信息通信发展司司长闻库表示，不了 …

Update from May 16:

Many of the limitations above have already been addressed.

Last night, Nyall has recorded a one hour tutorial on this new feature, enjoy:

Mapping spatial decision patterns, such as election results, is always a hot topic. That’s why we decided to include a recipe for election maps in our 老王vpm2.2.8下载安卓版 books. What’s new is that this recipe is now available as a free video tutorial recorded by Oliver Burdekin:

This video is just one of many recently published video tutorials that have been created by QGIS community members.

For example, Hans van der Kwast and Kurt Menke have recorded a 7-part series on QGIS for Hydrological Applications:

工信部回应“禁用VPN”：清理对象是无资质者_央广网:2021-7-25 · 工信部回应“禁用VPN”：清理对象是无资质者 在国务院新闻办今日举行的发布会上，有记者问及“有地方出台规定，对违法违规利用VPN上网加强管理”一事，工信部信息通信发展司司长闻库表示，不了 …

For the Pythonically inclined among you, there is also a new version of Python in QGIS on the Automating GIS-processes channel:

This post introduces Holoviz Panel, a library that makes it possible to create really quick dashboards in notebook environments as well as more sophisticated custom interactive web apps and dashboards.

The following example shows how to use Panel to explore a dataset (a trajectory collection in this case) and different parameter settings (relating to trajectory generalization). All the Panel code we need is a dict that defines the parameters that we want to explore. Then we can use Panel’s interact function to automatically generate a dashboard for our custom plotting function:

import panel as pn

kw = dict(traj_id=(1, len(traj_collection)), 
          tolerance=(10, 100, 10), 
          generalizer=['douglas-peucker', 'min-distance'])
pn.interact(plot_generalized, **kw)

Click to view the resulting dashboard in full resolution:

The plotting function uses the parameters to generate a Holoviews plot. First it fetches a specific trajectory from the trajectory collection. Then it generalizes the trajectory using the specified parameter settings. As you can see, we can easily combine maps and other plots to visualize different aspects of the data:

def plot_generalized(traj_id=1, tolerance=10, generalizer='douglas-peucker'):
  my_traj = traj_collection.get_trajectory(traj_id).to_crs(CRS(4088))
  if generalizer=='douglas-peucker':
    generalized = mpd.DouglasPeuckerGeneralizer(my_traj).generalize(tolerance)
  else:
    generalized = mpd.MinDistanceGeneralizer(my_traj).generalize(tolerance)
  generalized.add_speed(overwrite=True)
  return ( 
    generalized.hvplot(
      title='Trajectory {} (tolerance={})'.format(my_traj.id, tolerance), 
      c='speed', cmap='Viridis', colorbar=True, clim=(0,20), 
      line_width=10, width=500, height=500) + 
    generalized.df['speed'].hvplot.hist(
      title='Speed histogram', width=300, height=500) 
    )

Trajectory collections and generalization functions used in this example are part of the MovingPandas library. If you are interested in movement data analysis, you should check it out! You can find this example notebook in the MovingPandas tutorial section.

MovingPandas has come a long way since 2018 when I started to experiment with GeoPandas for trajectory data handling.

This week, MovingPandas passed peer review and was approved for pyOpenSci. This technical review process was extremely helpful in ensuring code, project, and documentation quality. I would strongly recommend it to everyone working on new data science libraries!

The lastest v0.3 release is now available from conda-forge.

All tutorials are available on MyBinder

New features include:

• Support for GeoPandas 0.7
• Trajectory collection aggregation functions to generate flow maps