提交 c5c549bc authored 作者: Luciano Paz's avatar Luciano Paz 提交者: Thomas Wiecki

Added support for the axis parameter of numpy.unique (#6644)

* Added support for the axis parameter of numpy.unique, which became available since numpy.__version__ 1.3.0. * Fixed doc example. Also fixed typo of the numpy version which added the axis parameter. The version that added axis was 1.13.0, not 1.3.0. * Made setup.py, conda and travis require numpy >= 1.13.0 in order to get numpy.unique with axis support. Changed test_Unique to test for axis=0 and axis=-1 parameter values. Fixed Unique.infer_shape bugs when axis is not None. * Updating scipy version to 0.19.1 * Updated miniconda version. * Fixed Unique Op doc typo. Split tests for Unique axis parameter. Success or failure now depends on numpy version. Added numpy version 1.13.0 and 1.9.1 as test cases in travisCI. Reverted required numpy version down to 1.9.1 in setup.py and conda. * Fixed Miniconda version to Miniconda2. Changed python3.6 to 3.7 in travisCI tests. * Fixed Unique Op doc for non axis handling. Changed python 3.7 to 3.6 in travisCI. * Added export MKL_THREADING_LAYER=GNU to travis script. This seems to be necessary because the updated miniconda2-4.5.11 installs 2018 MKL. * Fixed doc example when numpy version is < 1.13.0. Changed python 2.7 environment's scipy and libgfortran versions to avoid a build conflict in travisCI. Changed conda python 3.6 environment versions to avoid a build conflict in travisCI. * Updated numpy 1.13.0 to 1.13.1 in order to have numpy built with mkl support. * Fixed flake8 formating error. * Removed Unique examples in doc with axis input. These raised errors with sphinx, and conditional skiping was added in sphinx1.8, which is a higher version than the ones we test with. * Changed test_basic::test_grad `class O` to `class Obj1` to try to prevent flake8 ambiguous class definition E742 format error in python3.6 environment. * Hoping this makes sphinx not interpret 'py_' as a target name, which it does not know... * Changed Sphinx version down to 1.5.1. I had mistakenly thought that sphinx was responsible for downgrading numpy not to use mkl. * Fixed bug in tests. Instead of passing a function object to assertRaises, I was evaluating the function and passing its result. * Reduced the ammount of travis builds. * Removed trailing `<<: *normaltest` from .travis.yml. * All tests in `test_Unique_axis`, except `test_op`, now raise a `SkipTest` exception when the numpy version is lower than 1.13 * Implemented @nouiz's suggestions.
上级 65fefc3a
......@@ -39,7 +39,7 @@ install:
- pip install flake8-future-import parameterized sphinx_rtd_theme
# nose-exclude plugin allow us to tell nosetests to exclude folder with --exclude-dir=path/to/directory.
- pip install nose-exclude nose-timer
- conda install --yes -q scipy=0.14.0 # Try to reinstall it to fix the problem
- if [[ $NUMPY_VERSION == '1.13.1' ]]; then conda install --yes -q scipy=0.19.1; else conda install --yes -q scipy=0.14; fi # Try to reinstall it to fix the problem
jobs:
include:
......@@ -47,62 +47,75 @@ jobs:
- &doctest
stage: doc
python: "2.7"
env: DOC=1 PART="theano/tests/test_flake8.py"
env: NUMPY_VERSION=1.9.1 DOC=1 PART="theano/tests/test_flake8.py"
# re-use prototype, changing the Python version
- <<: *doctest
python: "2.7"
env: NUMPY_VERSION=1.13.1 DOC=1 PART="theano/tests/test_flake8.py"
- <<: *doctest
python: "3.4"
env: NUMPY_VERSION=1.9.1 DOC=1 PART="theano/tests/test_flake8.py"
- <<: *doctest
python: "3.6"
env: NUMPY_VERSION=1.13.1 DOC=1 PART="theano/tests/test_flake8.py"
- &normaltest
stage: test
python: "2.7"
env: PART="theano/compat theano/compile theano/d3viz theano/gof theano/misc theano/sandbox theano/scalar theano/scan_module theano/tests -e test_flake8.py theano/typed_list"
env: NUMPY_VERSION=1.9.1 PART="theano/compat theano/compile theano/d3viz theano/gof theano/misc theano/sandbox theano/scalar theano/scan_module theano/tests -e test_flake8.py theano/typed_list"
- <<: *normaltest
python: "3.4"
env: PART="theano/compat theano/compile theano/d3viz theano/gof theano/misc theano/sandbox theano/scalar theano/scan_module theano/tests -e test_flake8.py theano/typed_list"
env: NUMPY_VERSION=1.9.1 PART="theano/compat theano/compile theano/d3viz theano/gof theano/misc theano/sandbox theano/scalar theano/scan_module theano/tests -e test_flake8.py theano/typed_list"
- <<: *normaltest
env: PART="theano/sparse theano/tensor --exclude-test=theano.tensor.tests.test_basic --exclude-test=theano.tensor.tests.test_elemwise --exclude-test=theano.tensor.tests.test_opt --exclude-dir=theano/tensor/nnet"
env: NUMPY_VERSION=1.9.1 PART="theano/sparse theano/tensor --exclude-test=theano.tensor.tests.test_basic --exclude-test=theano.tensor.tests.test_elemwise --exclude-test=theano.tensor.tests.test_opt --exclude-dir=theano/tensor/nnet"
- <<: *normaltest
env: NUMPY_VERSION=1.13.1 PART="theano/sparse theano/tensor --exclude-test=theano.tensor.tests.test_basic --exclude-test=theano.tensor.tests.test_elemwise --exclude-test=theano.tensor.tests.test_opt --exclude-dir=theano/tensor/nnet"
- <<: *normaltest
python: "3.4"
env: PART="theano/sparse theano/tensor --exclude-test=theano.tensor.tests.test_basic --exclude-test=theano.tensor.tests.test_elemwise --exclude-test=theano.tensor.tests.test_opt --exclude-dir=theano/tensor/nnet"
env: NUMPY_VERSION=1.9.1 PART="theano/sparse theano/tensor --exclude-test=theano.tensor.tests.test_basic --exclude-test=theano.tensor.tests.test_elemwise --exclude-test=theano.tensor.tests.test_opt --exclude-dir=theano/tensor/nnet"
- <<: *normaltest
python: "3.6"
env: NUMPY_VERSION=1.13.1 PART="theano/sparse theano/tensor --exclude-test=theano.tensor.tests.test_basic --exclude-test=theano.tensor.tests.test_elemwise --exclude-test=theano.tensor.tests.test_opt --exclude-dir=theano/tensor/nnet"
- <<: *normaltest
env: PART="theano/tensor/tests/test_basic.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/tests/test_basic.py"
- <<: *normaltest
python: "3.4"
env: PART="theano/tensor/tests/test_basic.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/tests/test_basic.py"
- <<: *normaltest
env: PART="theano/tensor/tests/test_elemwise.py theano/tensor/tests/test_opt.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/tests/test_elemwise.py theano/tensor/tests/test_opt.py"
- <<: *normaltest
python: "3.4"
env: PART="theano/tensor/tests/test_elemwise.py theano/tensor/tests/test_opt.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/tests/test_elemwise.py theano/tensor/tests/test_opt.py"
- <<: *normaltest
env: PART="theano/tensor/nnet -e test_abstract_conv.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/nnet -e test_abstract_conv.py"
- <<: *normaltest
python: "3.4"
env: PART="theano/tensor/nnet -e test_abstract_conv.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/nnet -e test_abstract_conv.py"
- <<: *normaltest
env: PART="theano/tensor/nnet/tests/test_abstract_conv.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/nnet/tests/test_abstract_conv.py"
- <<: *normaltest
python: "3.4"
env: PART="theano/tensor/nnet/tests/test_abstract_conv.py"
env: NUMPY_VERSION=1.9.1 PART="theano/tensor/nnet/tests/test_abstract_conv.py"
- <<: *normaltest
env: FAST_COMPILE=1 FLOAT32=1 PART="theano -e test_flake8.py --exclude-dir=theano/tensor/nnet --exclude-dir=theano/tensor/signal"
env: NUMPY_VERSION=1.9.1 FAST_COMPILE=1 FLOAT32=1 PART="theano -e test_flake8.py --exclude-dir=theano/tensor/nnet --exclude-dir=theano/tensor/signal"
- <<: *normaltest
python: "3.4"
env: FAST_COMPILE=1 PART="theano -e test_flake8.py --exclude-dir=theano/tensor/nnet --exclude-dir=theano/tensor/signal"
env: NUMPY_VERSION=1.9.1 FAST_COMPILE=1 PART="theano -e test_flake8.py --exclude-dir=theano/tensor/nnet --exclude-dir=theano/tensor/signal"
- <<: *normaltest
env: FAST_COMPILE=1 FLOAT32=1 PART="theano/tensor/nnet"
env: NUMPY_VERSION=1.9.1 FAST_COMPILE=1 FLOAT32=1 PART="theano/tensor/nnet"
- <<: *normaltest
python: "3.4"
env: FAST_COMPILE=1 PART="theano/tensor/nnet"
env: NUMPY_VERSION=1.9.1 FAST_COMPILE=1 PART="theano/tensor/nnet"
- <<: *normaltest
env: FAST_COMPILE=1 FLOAT32=1 PART="theano/tensor/signal"
env: NUMPY_VERSION=1.9.1 FAST_COMPILE=1 FLOAT32=1 PART="theano/tensor/signal"
- <<: *normaltest
python: "3.4"
env: FAST_COMPILE=1 PART="theano/tensor/signal"
env: NUMPY_VERSION=1.9.1 FAST_COMPILE=1 PART="theano/tensor/signal"
script:
- if [[ $FAST_COMPILE == "1" ]]; then export THEANO_FLAGS=$THEANO_FLAGS,mode=FAST_COMPILE; fi
- if [[ $FLOAT32 == "1" ]]; then export THEANO_FLAGS=$THEANO_FLAGS,floatX=float32; fi
- export THEANO_FLAGS=$THEANO_FLAGS,warn.ignore_bug_before=all,on_opt_error=raise,on_shape_error=raise,gcc.cxxflags=-pipe
- export MKL_THREADING_LAYER=GNU
- export MKL_NUM_THREADS=1
- export OMP_NUM_THREADS=1
- python --version
......@@ -112,6 +125,7 @@ script:
- ulimit -a
- echo "$PART"
# Print information to help debug problems
- python -c 'import numpy; print(numpy.__version__)'
- python -c 'import theano; print(theano.__version__)'
- python -c 'import theano; print(theano.config.__str__(print_doc=False))'
- python -c 'import theano; assert(theano.config.blas.ldflags != "")'
......
......@@ -7,7 +7,7 @@ else
rm -rf $HOME/miniconda2
mkdir -p $HOME/download
if [[ -d $HOME/download/miniconda.sh ]] ; then rm -rf $HOME/download/miniconda.sh ; fi
wget -c https://repo.continuum.io/miniconda/Miniconda2-4.1.11-Linux-x86_64.sh -O $HOME/download/miniconda.sh
wget -c https://repo.continuum.io/miniconda/Miniconda2-4.5.11-Linux-x86_64.sh -O $HOME/download/miniconda.sh
chmod +x $HOME/download/miniconda.sh
$HOME/download/miniconda.sh -b
fi
......@@ -6,9 +6,12 @@ else
echo "Creating pyenv."
if [[ $TRAVIS_PYTHON_VERSION == '2.7' ]]; then conda create --yes -q -n pyenv python=2.7 ; fi
if [[ $TRAVIS_PYTHON_VERSION == '3.4' ]]; then conda create --yes -q -n pyenv python=3.4 ; fi
if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then conda create --yes -q -n pyenv python=3.6 ; fi
fi
source activate pyenv
if [[ $TRAVIS_PYTHON_VERSION == '2.7' ]]; then conda install --yes -q mkl numpy=1.9.1 scipy=0.14.0 nose=1.3.0 pip flake8=2.3 six=1.9.0 pep8=1.6.2 pyflakes=0.8.1 sphinx=1.5.1 mkl-service libgfortran=1 graphviz; fi
if [[ $TRAVIS_PYTHON_VERSION == '3.4' ]]; then conda install --yes -q mkl numpy=1.9.1 scipy=0.14.0 nose=1.3.4 pip flake8=2.3 six=1.9.0 pep8=1.6.2 pyflakes=0.8.1 sphinx=1.5.1 mkl-service libgfortran=1 graphviz; fi
if [[ $TRAVIS_PYTHON_VERSION == '2.7' && $NUMPY_VERSION == '1.9.1' ]]; then conda install --yes -q mkl numpy=1.9.1 scipy=0.14.0 nose=1.3.0 pip flake8=2.3 six=1.9.0 pep8=1.6.2 pyflakes=0.8.1 sphinx=1.5.1 mkl-service libgfortran=1 graphviz; fi
if [[ $TRAVIS_PYTHON_VERSION == '2.7' && $NUMPY_VERSION == '1.13.1' ]]; then conda install --yes -q mkl numpy=1.13.1 scipy=0.19.1 nose=1.3.0 pip flake8=2.3 six=1.9.0 pep8=1.6.2 pyflakes=0.8.1 sphinx=1.5.1 mkl-service libgfortran=3 graphviz; fi
if [[ $TRAVIS_PYTHON_VERSION == '3.4' && $NUMPY_VERSION == '1.9.1' ]]; then conda install --yes -q mkl numpy=1.9.1 scipy=0.14.0 nose=1.3.4 pip flake8=2.3 six=1.9.0 pep8=1.6.2 pyflakes=0.8.1 sphinx=1.5.1 mkl-service libgfortran=1 graphviz; fi
if [[ $TRAVIS_PYTHON_VERSION == '3.6' && $NUMPY_VERSION == '1.13.1' ]]; then conda install --yes -q mkl numpy=1.13.1 scipy=0.19.1 nose=1.3.7 pip flake8=3.5 six=1.11.0 pep8=1.7.1 pyflakes=1.6.0 sphinx=1.5.1 mkl-service libgfortran=3 graphviz; fi
source deactivate
......@@ -1127,23 +1127,39 @@ class Unique(theano.Op):
"""
__props__ = ("return_index", "return_inverse", "return_counts")
__props__ = ("return_index", "return_inverse", "return_counts",
"axis")
def __init__(self, return_index=False, return_inverse=False,
return_counts=False):
return_counts=False, axis=None):
self.return_index = return_index
self.return_inverse = return_inverse
self.return_counts = return_counts
self.axis = axis
numpy_ver = [int(n) for n in np.__version__.split('.')[:2]]
if self.return_counts and bool(numpy_ver < [1, 9]):
if self.axis is not None and bool(numpy_ver < [1, 13]):
raise RuntimeError(
"Numpy version = " + np.__version__ +
". Option 'return_counts=True' works starting"
" from version 1.9.0.")
". Option 'axis={}' works starting"
" from version 1.13.0.".format(axis))
def make_node(self, x):
x = basic.as_tensor_variable(x)
outputs = [basic.TensorType(broadcastable=[False], dtype=x.dtype)()]
self_axis = self.axis
if self_axis is None:
broadcastable = [False]
else:
if self_axis < 0:
self_axis += len(x.broadcastable)
if self_axis < 0 or self_axis >= len(x.broadcastable):
raise RuntimeError(
"Unique axis `{}` is outside of input ndim = "
"{}.".format(self.axis, len(x.broadcastable))
)
broadcastable = [b if axis != self_axis else False
for axis, b in enumerate(x.broadcastable)]
outputs = [basic.TensorType(broadcastable=broadcastable,
dtype=x.dtype)()]
typ = basic.TensorType(broadcastable=[False], dtype='int64')
if self.return_index:
outputs.append(typ())
......@@ -1163,6 +1179,8 @@ class Unique(theano.Op):
param['return_inverse'] = True
if self.return_counts:
param['return_counts'] = True
if self.axis is not None:
param['axis'] = self.axis
outs = np.unique(x, **param)
if ((not self.return_inverse) and
(not self.return_index) and
......@@ -1174,8 +1192,24 @@ class Unique(theano.Op):
def infer_shape(self, node, i0_shapes):
ret = node.fgraph.shape_feature.default_infer_shape(node, i0_shapes)
if self.axis is not None:
self_axis = self.axis
ndim = len(i0_shapes[0])
if self_axis < 0:
self_axis += ndim
if self_axis < 0 or self_axis >= ndim:
raise RuntimeError(
"Unique axis `{}` is outside of input ndim = "
"{}.".format(self.axis, ndim)
)
ret[0] = tuple([node.fgraph.shape_feature.shape_ir(i,
node.outputs[0])
for i in xrange(ndim)])
if self.return_inverse:
shape = (basic.prod(i0_shapes[0]), )
if self.axis is None:
shape = (basic.prod(i0_shapes[0]), )
else:
shape = (i0_shapes[0][self_axis], )
if self.return_index:
ret[2] = shape
return ret
......@@ -1183,6 +1217,13 @@ class Unique(theano.Op):
return ret
return ret
def __setstate__(self, state):
self.__dict__.update(state)
# For backwards compatibility with pickled instances of Unique that
# did not have the axis parameter specified
if 'axis' not in state:
self.axis = None
class UnravelIndex(gof.Op):
__props__ = ('ndim', 'order')
......
......@@ -5342,7 +5342,7 @@ class T_scalarfromtensor(unittest.TestCase):
class test_grad(unittest.TestCase):
class O(gof.op.Op):
class Obj1(gof.op.Op):
def __init__(self):
self.gval0 = scalar('e')
self.gval1 = scalar('f')
......@@ -5359,13 +5359,13 @@ class test_grad(unittest.TestCase):
def test_1param(self):
# grad: Test passing a single variable param
o = test_grad.O()
o = test_grad.Obj1()
a1 = o.make_node()
self.assertTrue(o.gval0 is tensor.grad(a1.outputs[0], a1.inputs[0]))
def test_Nparam(self):
# grad: Test passing multiple variable params
o = test_grad.O()
o = test_grad.Obj1()
a1 = o.make_node()
g0, g1 = grad(a1.outputs[0], a1.inputs)
g0.name = None
......@@ -5393,7 +5393,7 @@ class test_grad(unittest.TestCase):
def test_1None_rval(self):
# grad: Test returning a single zero value from grad
o = test_grad.O()
o = test_grad.Obj1()
a1 = o.make_node()
g = grad(a1.outputs[0], a1.outputs[1],
disconnected_inputs='ignore')
......@@ -5403,7 +5403,7 @@ class test_grad(unittest.TestCase):
def test_NNone_rval(self):
# grad: Test returning some zero value from grad
o = test_grad.O()
o = test_grad.Obj1()
a1 = o.make_node()
g0, g1, g2 = grad(a1.outputs[0], a1.inputs + [scalar('z')],
disconnected_inputs='ignore')
......
......@@ -705,7 +705,7 @@ class test_Unique(utt.InferShapeTester):
# Done by using the op and checking that it returns the right answer.
x = theano.tensor.matrix()
inp = np.asarray([[2, 1], [3, 2], [2, 3]], dtype=config.floatX)
inp = np.asarray([[2, 1], [3, 2], [2, 1]], dtype=config.floatX)
list_outs_expected = [[np.unique(inp)],
np.unique(inp, True),
np.unique(inp, False, True),
......@@ -758,6 +758,133 @@ class test_Unique(utt.InferShapeTester):
self.op_class)
class test_Unique_axis(utt.InferShapeTester):
def setUp(self):
super(test_Unique_axis, self).setUp()
numpy_ver = tuple([int(n) for n in np.__version__.split('.')])
if numpy_ver >= (1, 13):
self.expect_success = True
else:
self.expect_success = False
self.ops_pars = [(tuple(), {'axis': 0}),
((True,), {'axis': 0}),
((False, True,), {'axis': 0}),
((True, True,), {'axis': 0}),
((False, False, True,), {'axis': 0}),
((True, False, True,), {'axis': 0}),
((False, True, True,), {'axis': 0}),
((True, True, True,), {'axis': 0}),
(tuple(), {'axis': -1}),
((True,), {'axis': -1}),
((False, True,), {'axis': -1}),
((True, True,), {'axis': -1}),
((False, False, True,), {'axis': -1}),
((True, False, True,), {'axis': -1}),
((False, True, True,), {'axis': -1}),
((True, True, True,), {'axis': -1})]
self.op_class = Unique
def test_op(self):
if self.expect_success:
for args, kwargs in self.ops_pars:
op = self.op_class(*args, **kwargs)
self.assertTrue(isinstance(op, self.op_class))
else:
for args, kwargs in self.ops_pars:
def func():
return self.op_class(*args, **kwargs)
self.assertRaises(RuntimeError, func)
def test_basic_vector(self):
if not self.expect_success:
raise utt.SkipTest('Requires numpy >= 1.13')
# Basic test for a vector.
# Done by using the op and checking that it returns the right
# answer.
x = theano.tensor.vector()
ops = [self.op_class(*args, **kwargs) for args, kwargs in
self.ops_pars]
inp = np.asarray([2, 1, 3, 2], dtype=config.floatX)
list_outs_expected = [[np.unique(inp, **kwargs)] if len(args) == 0
else np.unique(inp, *args, **kwargs)
for args, kwargs in self.ops_pars]
for op, outs_expected in zip(ops, list_outs_expected):
f = theano.function(inputs=[x],
outputs=op(x, return_list=True))
outs = f(inp)
# Compare the result computed to the expected value.
for out, out_exp in zip(outs, outs_expected):
utt.assert_allclose(out, out_exp)
def test_basic_matrix(self):
if not self.expect_success:
raise utt.SkipTest('Requires numpy >= 1.13')
# Basic test for a matrix.
# Done by using the op and checking that it returns the right
# answer.
x = theano.tensor.matrix()
ops = [self.op_class(*args, **kwargs) for args, kwargs in
self.ops_pars]
inp = np.asarray([[2, 1], [3, 2], [2, 1]], dtype=config.floatX)
list_outs_expected = [[np.unique(inp, **kwargs)] if len(args) == 0
else np.unique(inp, *args, **kwargs)
for args, kwargs in self.ops_pars]
for op, outs_expected in zip(ops, list_outs_expected):
f = theano.function(inputs=[x],
outputs=op(x, return_list=True))
outs = f(inp)
# Compare the result computed to the expected value.
for out, out_exp in zip(outs, outs_expected):
utt.assert_allclose(out, out_exp)
def test_infer_shape_vector(self):
if not self.expect_success:
raise utt.SkipTest('Requires numpy >= 1.13')
# Testing the infer_shape with a vector.
x = theano.tensor.vector()
ops = [self.op_class(*args, **kwargs) for args, kwargs in
self.ops_pars]
for op in ops:
if not op.return_inverse:
continue
if op.return_index:
f = op(x)[2]
else:
f = op(x)[1]
self._compile_and_check([x],
[f],
[np.asarray(np.array([2, 1, 3, 2]),
dtype=config.floatX)],
self.op_class)
def test_infer_shape_matrix(self):
if not self.expect_success:
raise utt.SkipTest('Requires numpy >= 1.13')
# Testing the infer_shape with a matrix.
x = theano.tensor.matrix()
ops = [self.op_class(*args, **kwargs) for args, kwargs in
self.ops_pars]
for op in ops:
if not op.return_inverse:
continue
if op.return_index:
f = op(x)[2]
else:
f = op(x)[1]
self._compile_and_check([x],
[f],
[np.asarray(np.array([[2, 1], [3, 2], [2, 1]]),
dtype=config.floatX)],
self.op_class)
class test_unravel_index(utt.InferShapeTester):
def test_unravel_index(self):
def check(shape, index_ndim, order):
......
Markdown 格式
0%
您添加了 0 到此讨论。请谨慎行事。
请先完成此评论的编辑!
注册 或者 后发表评论