Pedro Miguel Marcos Joaquim

Leveraging Transient Resources for Incremental Graph Processing on Heterogeneous Infrastructures

Tese submetida para provas de mestrado em Engenharia Informática e de Computadores Instituto Superior Técnico, Universidade de Lisboa.


Incremental graph processing as emerged as a key technology with applications in many different areas. This task is typically executed in public cloud infrastructures that offer a wide variety of services. The heterogeneity of the provided resources and the different renting services available, such as reserved or transient-lived resources, offer a good opportunity to match cost with intended deployment characteris- tics. In this paper we present Hourglass, a deployment engine for existing incremental graph processing systems that aims at leveraging the existing heterogeneity in such deployment environments, as well as transient resource markets to significantly reduce costs. The system exploits features of the workload, such as the imbalance in vertex communication patterns, to do a resource-aware distribution of graph partitions among the existing machines. Hourglass constantly analyses market conditions to find the best possible deployment in specific moments in time. Knowledge about the computation being per- formed allows Hourglass to choose the right moment to transition between deployments and allow the underlying system to operate on a fraction of the cost of a system composed exclusively of reserved machines, with minimal impact on the system performance.


Leveraging Transient Resources for Incremental Graph Processing on Heterogeneous Infrastructures
Pedro Miguel Marcos Joaquim
MSc Thesis. Instituto Superior Técnico, Universidade de Lisboa.
November, 2017.
Available BibTeX, MSC Thesis, and extended abstract, and mid-term report.
Processamento Incremental de Grafos em Sistemas Heterogêneos.
P. Joaquim, M. Bravo, R. Rodrigues, M. Matos and L. Rodrigues.
Actas do nono Simpósio de Informática (Inforum), Lisboa, Portugal, Oct. 2017.
Available BibTeX, extended report (pdf).
Hourglass: Leveraging Transient Resources for Time-Constrained Graph Processing in the Cloud.
P. Joaquim, M. Bravo, L. Rodrigues, M. Matos
In Proceedings of the EuroSys 2019, Dresden, Germany, March 2019.

Luís Rodrigues