sparse-matrix

Iterating through a scipy.sparse vector (or matrix)

by

Edit: bbtrb’s method (using coo_matrix) is much faster than my original suggestion, using nonzero. Sven Marnach’s suggestion to use itertools.izip also improves the speed. Current fastest is using_tocoo_izip: import scipy.sparse import random import itertools def using_nonzero(x): rows,cols = x.nonzero() for row,col in zip(rows,cols): ((row,col), x[row,col]) def using_coo(x): cx = scipy.sparse.coo_matrix(x) for i,j,v in zip(cx.row, cx.col, … Read more

Directly use Intel mkl library on Scipy sparse matrix to calculate A dot A.T with less memory

by

Look at the Python code for the scipy sparse product. Notice that it calls the compiled code in 2 passes. It looks like the mkl code does the same thing https://software.intel.com/en-us/node/468640 If request=1, the routine computes only values of the array ic of length m + 1, the memory for this array must be allocated … Read more

Concatenate sparse matrices in Python using SciPy/Numpy

by

You can use the scipy.sparse.hstack to concatenate sparse matrices with the same number of rows (horizontal concatenation): from scipy.sparse import hstack hstack((X, X2)) Similarly, you can use scipy.sparse.vstack to concatenate sparse matrices with the same number of columns (vertical concatenation). Using numpy.hstack or numpy.vstack will create an array with two sparse matrix objects.

Pandas sparse dataFrame to sparse matrix, without generating a dense matrix in memory

by

Pandas 0.20.0+: As of pandas version 0.20.0, released May 5, 2017, there is a one-liner for this: from scipy import sparse def sparse_df_to_csr(df): return sparse.csr_matrix(df.to_coo()) This uses the new to_coo() method. Earlier Versions: Building on Victor May’s answer, here’s a slightly faster implementation, but it only works if the entire SparseDataFrame is sparse with all … Read more

Populate a Pandas SparseDataFrame from a SciPy Sparse Matrix

by

A direct conversion is not supported ATM. Contributions are welcome! Try this, should be ok on memory as the SpareSeries is much like a csc_matrix (for 1 column) and pretty space efficient In [37]: col = np.array([0,0,1,2,2,2]) In [38]: data = np.array([1,2,3,4,5,6],dtype=”float64″) In [39]: m = csc_matrix( (data,(row,col)), shape=(3,3) ) In [40]: m Out[40]: <3×3 … Read more

Sparse matrix slicing using list of int

by

I think I’ve recreated the csr row indexing with: def extractor(indices, N): indptr=np.arange(len(indices)+1) data=np.ones(len(indices)) shape=(len(indices),N) return sparse.csr_matrix((data,indices,indptr), shape=shape) Testing on a csr I had hanging around: In [185]: M Out[185]: <30×40 sparse matrix of type ‘<class ‘numpy.float64′>’ with 76 stored elements in Compressed Sparse Row format> In [186]: indices=np.r_[0:20] In [187]: M[indices,:] Out[187]: <20×40 sparse … Read more

Are Javascript arrays sparse?

by

Yes, they are. They are actually hash tables internally, so you can use not only large integers but also strings, floats, or other objects. All keys get converted to strings via toString() before being added to the hash. You can confirm this with some test code: <script> var array = []; array[0] = “zero”; array[new … Read more

tech