We are releasing two calls alongside this paper to encourage, increase, and broaden the reach of scientific interactions and collaborations. The two calls are an invitation for fellow researchers to address two questions that are not yet sufficiently answered by this work:
- What are plausible explanations of the effectiveness of ASH, a simple activation pruning and readjusting technique, on ID and OOD tasks?
- Are there other research domains, application areas, topics and tasks where ASH (or a similar procedure) is applicable, and what are the findings?
Answers to these calls will be carefully reviewed and selectively included in future versions of this paper, where individual contributors will be invited to collaborate.
For each call we provide possible directions to explore the answer, however, we encourage novel quests beyond what's suggested below.
Call for explanation. A possible explanation of the effectiveness of ASH is that our overparameterized networks likely overdo representation learning—generating features for data that are largely redundant for the optimization task at hand. It is both an advantage and a peril: on the one hand the representation is less likely to overfit to a single task and might retain more potential to generalize, but on the other hand it serves a poorer discriminator between data seen and unseen.
Call for validation in other fields. We think any domains that use a deep neural network (or a similar intelligent system) to learn representations of data when optimizing for a training task would be fertile ground for validating ASH. A straightforward domain is natural language processing, where pretrained language models are often adapted for downstream tasks. Are native representations learned with those large language models simplifiable? Would reshaping of activations (in the case of transformer-based language models, activations can be keys, values or queries) enhance or damage the performance?
We are still working to set up a proper portal for submitting, reviewing and discussing answers to both calls. (ETA: likely after the ICLR deadline.) In the meantime, feel free to email firstname.lastname@example.org to start a conversation.